* [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-09 13:14 Chris Snook 2007-08-09 12:41 ` Arnd Bergmann 2007-08-14 22:31 ` Christoph Lameter 0 siblings, 2 replies; 333+ messages in thread From: Chris Snook @ 2007-08-09 13:14 UTC (permalink / raw) To: linux-kernel, linux-arch, torvalds Cc: netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl As recent discussions[1], and bugs[2] have shown, there is a great deal of confusion about the expected behavior of atomic_read(), compounded by the fact that it is not the same on all architectures. Since users expect calls to atomic_read() to actually perform a read, it is not desirable to allow the compiler to optimize this away. Requiring the use of barrier() in this case is inefficient, since we only want to re-load the atomic_t variable, not everything else in scope. This patchset makes the behavior of atomic_read uniform by removing the volatile keyword from all atomic_t and atomic64_t definitions that currently have it, and instead explicitly casts the variable as volatile in atomic_read(). This leaves little room for creative optimization by the compiler, and is in keeping with the principles behind "volatile considered harmful". Busy-waiters should still use cpu_relax(), but fast paths may be able to reduce their use of barrier() between some atomic_read() calls. -- Chris 1) http://lkml.org/lkml/2007/7/1/52 2) http://lkml.org/lkml/2007/8/8/122 ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-09 13:14 [PATCH 0/24] make atomic_read() behave consistently across all architectures Chris Snook @ 2007-08-09 12:41 ` Arnd Bergmann 2007-08-09 14:29 ` Chris Snook 2007-08-14 22:31 ` Christoph Lameter 1 sibling, 1 reply; 333+ messages in thread From: Arnd Bergmann @ 2007-08-09 12:41 UTC (permalink / raw) To: Chris Snook Cc: linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl On Thursday 09 August 2007, Chris Snook wrote: > This patchset makes the behavior of atomic_read uniform by removing the > volatile keyword from all atomic_t and atomic64_t definitions that currently > have it, and instead explicitly casts the variable as volatile in > atomic_read(). This leaves little room for creative optimization by the > compiler, and is in keeping with the principles behind "volatile considered > harmful". > Just an idea: since all architectures already include asm-generic/atomic.h, why not move the definitions of atomic_t and atomic64_t, as well as anything that does not involve architecture specific inline assembly into the generic header? Arnd <>< ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-09 12:41 ` Arnd Bergmann @ 2007-08-09 14:29 ` Chris Snook 2007-08-09 15:30 ` Arnd Bergmann 0 siblings, 1 reply; 333+ messages in thread From: Chris Snook @ 2007-08-09 14:29 UTC (permalink / raw) To: Arnd Bergmann Cc: linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl Arnd Bergmann wrote: > On Thursday 09 August 2007, Chris Snook wrote: >> This patchset makes the behavior of atomic_read uniform by removing the >> volatile keyword from all atomic_t and atomic64_t definitions that currently >> have it, and instead explicitly casts the variable as volatile in >> atomic_read(). This leaves little room for creative optimization by the >> compiler, and is in keeping with the principles behind "volatile considered >> harmful". >> > > Just an idea: since all architectures already include asm-generic/atomic.h, > why not move the definitions of atomic_t and atomic64_t, as well as anything > that does not involve architecture specific inline assembly into the generic > header? > > Arnd <>< a) chicken and egg: asm-generic/atomic.h depends on definitions in asm/atomic.h If you can find a way to reshuffle the code and make it simpler, I personally am all for it. I'm skeptical that you'll get much to show for the effort. b) The definitions aren't precisely identical between all architectures, so it would be a mess of special cases, which gets us right back to where we are now. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-09 14:29 ` Chris Snook @ 2007-08-09 15:30 ` Arnd Bergmann 0 siblings, 0 replies; 333+ messages in thread From: Arnd Bergmann @ 2007-08-09 15:30 UTC (permalink / raw) To: Chris Snook Cc: linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl On Thursday 09 August 2007, Chris Snook wrote: > a) chicken and egg: asm-generic/atomic.h depends on definitions in asm/atomic.h Ok, I see. > If you can find a way to reshuffle the code and make it simpler, I personally am > all for it. I'm skeptical that you'll get much to show for the effort. I guess it could be done using more macros or new headers, but I don't see a way that would actually improve the situation. > b) The definitions aren't precisely identical between all architectures, so it > would be a mess of special cases, which gets us right back to where we are now. Why are they not identical? Anything beyond the 32/64 bit difference should be the same afaics, or it might cause more bugs. Arnd <>< ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-09 13:14 [PATCH 0/24] make atomic_read() behave consistently across all architectures Chris Snook 2007-08-09 12:41 ` Arnd Bergmann @ 2007-08-14 22:31 ` Christoph Lameter 2007-08-14 22:45 ` Chris Snook 2007-08-14 23:08 ` Satyam Sharma 1 sibling, 2 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-14 22:31 UTC (permalink / raw) To: Chris Snook Cc: linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl On Thu, 9 Aug 2007, Chris Snook wrote: > This patchset makes the behavior of atomic_read uniform by removing the > volatile keyword from all atomic_t and atomic64_t definitions that currently > have it, and instead explicitly casts the variable as volatile in > atomic_read(). This leaves little room for creative optimization by the > compiler, and is in keeping with the principles behind "volatile considered > harmful". volatile is generally harmful even in atomic_read(). Barriers control visibility and AFAICT things are fine. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 22:31 ` Christoph Lameter @ 2007-08-14 22:45 ` Chris Snook 2007-08-14 22:51 ` Christoph Lameter 2007-08-14 23:08 ` Satyam Sharma 1 sibling, 1 reply; 333+ messages in thread From: Chris Snook @ 2007-08-14 22:45 UTC (permalink / raw) To: Christoph Lameter Cc: linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl Christoph Lameter wrote: > On Thu, 9 Aug 2007, Chris Snook wrote: > >> This patchset makes the behavior of atomic_read uniform by removing the >> volatile keyword from all atomic_t and atomic64_t definitions that currently >> have it, and instead explicitly casts the variable as volatile in >> atomic_read(). This leaves little room for creative optimization by the >> compiler, and is in keeping with the principles behind "volatile considered >> harmful". > > volatile is generally harmful even in atomic_read(). Barriers control > visibility and AFAICT things are fine. But barriers force a flush of *everything* in scope, which we generally don't want. On the other hand, we pretty much always want to flush atomic_* operations. One way or another, we should be restricting the volatile behavior to the thing that needs it. On most architectures, this patch set just moves that from the declaration, where it is considered harmful, to the use, where it is considered an occasional necessary evil. See the resubmitted patchset, which also puts a cast in the atomic[64]_set operations. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 22:45 ` Chris Snook @ 2007-08-14 22:51 ` Christoph Lameter 0 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-14 22:51 UTC (permalink / raw) To: Chris Snook Cc: linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl On Tue, 14 Aug 2007, Chris Snook wrote: > But barriers force a flush of *everything* in scope, which we generally don't > want. On the other hand, we pretty much always want to flush atomic_* > operations. One way or another, we should be restricting the volatile > behavior to the thing that needs it. On most architectures, this patch set > just moves that from the declaration, where it is considered harmful, to the > use, where it is considered an occasional necessary evil. Then we would need atomic_read() and atomic_read_volatile() atomic_read_volatile() would imply an object sized memory barrier before and after? ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 22:31 ` Christoph Lameter 2007-08-14 22:45 ` Chris Snook @ 2007-08-14 23:08 ` Satyam Sharma 2007-08-14 23:04 ` Chris Snook ` (2 more replies) 1 sibling, 3 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-14 23:08 UTC (permalink / raw) To: Christoph Lameter Cc: Chris Snook, Linux Kernel Mailing List, linux-arch, torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Tue, 14 Aug 2007, Christoph Lameter wrote: > On Thu, 9 Aug 2007, Chris Snook wrote: > > > This patchset makes the behavior of atomic_read uniform by removing the > > volatile keyword from all atomic_t and atomic64_t definitions that currently > > have it, and instead explicitly casts the variable as volatile in > > atomic_read(). This leaves little room for creative optimization by the > > compiler, and is in keeping with the principles behind "volatile considered > > harmful". > > volatile is generally harmful even in atomic_read(). Barriers control > visibility and AFAICT things are fine. Frankly, I don't see the need for this series myself either. Personal opinion (others may differ), but I consider "volatile" to be a sad / unfortunate wart in C (numerous threads on this list and on the gcc lists/bugzilla over the years stand testimony to this) and if we _can_ steer clear of it, then why not -- why use this ill-defined primitive whose implementation has often differed over compiler versions and platforms? Granted, barrier() _is_ heavy-handed in that it makes the optimizer forget _everything_, but then somebody did post a forget() macro on this thread itself ... [ BTW, why do we want the compiler to not optimize atomic_read()'s in the first place? Atomic ops guarantee atomicity, which has nothing to do with "volatility" -- users that expect "volatility" from atomic ops are the ones who must be fixed instead, IMHO. ] ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 23:08 ` Satyam Sharma @ 2007-08-14 23:04 ` Chris Snook 2007-08-14 23:14 ` Christoph Lameter 2007-08-15 6:49 ` Herbert Xu 2007-08-14 23:26 ` Paul E. McKenney 2007-08-15 10:35 ` Stefan Richter 2 siblings, 2 replies; 333+ messages in thread From: Chris Snook @ 2007-08-14 23:04 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Linux Kernel Mailing List, linux-arch, torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > > On Tue, 14 Aug 2007, Christoph Lameter wrote: > >> On Thu, 9 Aug 2007, Chris Snook wrote: >> >>> This patchset makes the behavior of atomic_read uniform by removing the >>> volatile keyword from all atomic_t and atomic64_t definitions that currently >>> have it, and instead explicitly casts the variable as volatile in >>> atomic_read(). This leaves little room for creative optimization by the >>> compiler, and is in keeping with the principles behind "volatile considered >>> harmful". >> volatile is generally harmful even in atomic_read(). Barriers control >> visibility and AFAICT things are fine. > > Frankly, I don't see the need for this series myself either. Personal > opinion (others may differ), but I consider "volatile" to be a sad / > unfortunate wart in C (numerous threads on this list and on the gcc > lists/bugzilla over the years stand testimony to this) and if we _can_ > steer clear of it, then why not -- why use this ill-defined primitive > whose implementation has often differed over compiler versions and > platforms? Granted, barrier() _is_ heavy-handed in that it makes the > optimizer forget _everything_, but then somebody did post a forget() > macro on this thread itself ... > > [ BTW, why do we want the compiler to not optimize atomic_read()'s in > the first place? Atomic ops guarantee atomicity, which has nothing > to do with "volatility" -- users that expect "volatility" from > atomic ops are the ones who must be fixed instead, IMHO. ] Because atomic operations are generally used for synchronization, which requires volatile behavior. Most such codepaths currently use an inefficient barrier(). Some forget to and we get bugs, because people assume that atomic_read() actually reads something, and atomic_write() actually writes something. Worse, these are architecture-specific, even compiler version-specific bugs that are often difficult to track down. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 23:04 ` Chris Snook @ 2007-08-14 23:14 ` Christoph Lameter 2007-08-15 6:49 ` Herbert Xu 1 sibling, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-14 23:14 UTC (permalink / raw) To: Chris Snook Cc: Satyam Sharma, Linux Kernel Mailing List, linux-arch, torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Tue, 14 Aug 2007, Chris Snook wrote: > Because atomic operations are generally used for synchronization, which > requires volatile behavior. Most such codepaths currently use an inefficient > barrier(). Some forget to and we get bugs, because people assume that > atomic_read() actually reads something, and atomic_write() actually writes > something. Worse, these are architecture-specific, even compiler > version-specific bugs that are often difficult to track down. Looks like we need to have lock and unlock semantics? atomic_read() which has no barrier or volatile implications. atomic_read_for_lock Acquire semantics? atomic_read_for_unlock Release semantics? ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 23:04 ` Chris Snook @ 2007-08-15 6:49 ` Herbert Xu 2007-08-15 6:49 ` Herbert Xu 1 sibling, 0 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-15 6:49 UTC (permalink / raw) To: Chris Snook Cc: satyam, clameter, linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Chris Snook <csnook@redhat.com> wrote: > > Because atomic operations are generally used for synchronization, which requires > volatile behavior. Most such codepaths currently use an inefficient barrier(). > Some forget to and we get bugs, because people assume that atomic_read() > actually reads something, and atomic_write() actually writes something. Worse, > these are architecture-specific, even compiler version-specific bugs that are > often difficult to track down. I'm yet to see a single example from the current tree where this patch series is the correct solution. So far the only example has been a buggy piece of code which has since been fixed with a cpu_relax. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-15 6:49 ` Herbert Xu 0 siblings, 0 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-15 6:49 UTC (permalink / raw) To: Chris Snook Cc: satyam, clameter, linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Chris Snook <csnook@redhat.com> wrote: > > Because atomic operations are generally used for synchronization, which requires > volatile behavior. Most such codepaths currently use an inefficient barrier(). > Some forget to and we get bugs, because people assume that atomic_read() > actually reads something, and atomic_write() actually writes something. Worse, > these are architecture-specific, even compiler version-specific bugs that are > often difficult to track down. I'm yet to see a single example from the current tree where this patch series is the correct solution. So far the only example has been a buggy piece of code which has since been fixed with a cpu_relax. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 6:49 ` Herbert Xu (?) @ 2007-08-15 8:18 ` Heiko Carstens 2007-08-15 13:53 ` Stefan Richter 2007-08-16 0:39 ` [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() Satyam Sharma -1 siblings, 2 replies; 333+ messages in thread From: Heiko Carstens @ 2007-08-15 8:18 UTC (permalink / raw) To: Herbert Xu Cc: Chris Snook, satyam, clameter, linux-kernel, linux-arch, torvalds, netdev, akpm, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 02:49:03PM +0800, Herbert Xu wrote: > Chris Snook <csnook@redhat.com> wrote: > > > > Because atomic operations are generally used for synchronization, which requires > > volatile behavior. Most such codepaths currently use an inefficient barrier(). > > Some forget to and we get bugs, because people assume that atomic_read() > > actually reads something, and atomic_write() actually writes something. Worse, > > these are architecture-specific, even compiler version-specific bugs that are > > often difficult to track down. > > I'm yet to see a single example from the current tree where > this patch series is the correct solution. So far the only > example has been a buggy piece of code which has since been > fixed with a cpu_relax. Btw.: we still have include/asm-i386/mach-es7000/mach_wakecpu.h: while (!atomic_read(deassert)); include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert)); Looks like they need to be fixed as well. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 8:18 ` Heiko Carstens @ 2007-08-15 13:53 ` Stefan Richter 2007-08-15 14:35 ` Satyam Sharma 2007-08-16 0:39 ` [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() Satyam Sharma 1 sibling, 1 reply; 333+ messages in thread From: Stefan Richter @ 2007-08-15 13:53 UTC (permalink / raw) To: Heiko Carstens Cc: Herbert Xu, Chris Snook, satyam, clameter, linux-kernel, linux-arch, torvalds, netdev, akpm, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On 8/15/2007 10:18 AM, Heiko Carstens wrote: > On Wed, Aug 15, 2007 at 02:49:03PM +0800, Herbert Xu wrote: >> Chris Snook <csnook@redhat.com> wrote: >> > >> > Because atomic operations are generally used for synchronization, which requires >> > volatile behavior. Most such codepaths currently use an inefficient barrier(). >> > Some forget to and we get bugs, because people assume that atomic_read() >> > actually reads something, and atomic_write() actually writes something. Worse, >> > these are architecture-specific, even compiler version-specific bugs that are >> > often difficult to track down. >> >> I'm yet to see a single example from the current tree where >> this patch series is the correct solution. So far the only >> example has been a buggy piece of code which has since been >> fixed with a cpu_relax. > > Btw.: we still have > > include/asm-i386/mach-es7000/mach_wakecpu.h: while (!atomic_read(deassert)); > include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert)); > > Looks like they need to be fixed as well. I don't know if this here is affected: /* drivers/ieee1394/ieee1394_core.h */ static inline unsigned int get_hpsb_generation(struct hpsb_host *host) { return atomic_read(&host->generation); } /* drivers/ieee1394/nodemgr.c */ static int nodemgr_host_thread(void *__hi) { [...] for (;;) { [... sleep until bus reset event ...] /* Pause for 1/4 second in 1/16 second intervals, * to make sure things settle down. */ g = get_hpsb_generation(host); for (i = 0; i < 4 ; i++) { if (msleep_interruptible(63) || kthread_should_stop()) goto exit; /* Now get the generation in which the node ID's we collect * are valid. During the bus scan we will use this generation * for the read transactions, so that if another reset occurs * during the scan the transactions will fail instead of * returning bogus data. */ generation = get_hpsb_generation(host); /* If we get a reset before we are done waiting, then * start the waiting over again */ if (generation != g) g = generation, i = 0; } [... scan bus, using generation ...] } exit: [...] } -- Stefan Richter -=====-=-=== =--- -==== http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 13:53 ` Stefan Richter @ 2007-08-15 14:35 ` Satyam Sharma 2007-08-15 14:52 ` Herbert Xu 2007-08-15 19:58 ` Stefan Richter 0 siblings, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 14:35 UTC (permalink / raw) To: Stefan Richter Cc: Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Hi Stefan, On Wed, 15 Aug 2007, Stefan Richter wrote: > On 8/15/2007 10:18 AM, Heiko Carstens wrote: > > On Wed, Aug 15, 2007 at 02:49:03PM +0800, Herbert Xu wrote: > >> Chris Snook <csnook@redhat.com> wrote: > >> > > >> > Because atomic operations are generally used for synchronization, which requires > >> > volatile behavior. Most such codepaths currently use an inefficient barrier(). > >> > Some forget to and we get bugs, because people assume that atomic_read() > >> > actually reads something, and atomic_write() actually writes something. Worse, > >> > these are architecture-specific, even compiler version-specific bugs that are > >> > often difficult to track down. > >> > >> I'm yet to see a single example from the current tree where > >> this patch series is the correct solution. So far the only > >> example has been a buggy piece of code which has since been > >> fixed with a cpu_relax. > > > > Btw.: we still have > > > > include/asm-i386/mach-es7000/mach_wakecpu.h: while (!atomic_read(deassert)); > > include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert)); > > > > Looks like they need to be fixed as well. > > > I don't know if this here is affected: Yes, I think it is. You're clearly expecting the read to actually happen when you call get_hpsb_generation(). It's clearly not a busy-loop, so cpu_relax() sounds pointless / wrong solution for this case, so I'm now somewhat beginning to appreciate the motivation behind this series :-) But as I said, there are ways to achieve the same goals of this series without using "volatile". I think I'll submit a RFC/patch or two on this myself (will also fix the code pieces listed here). > /* drivers/ieee1394/ieee1394_core.h */ > static inline unsigned int get_hpsb_generation(struct hpsb_host *host) > { > return atomic_read(&host->generation); > } > > /* drivers/ieee1394/nodemgr.c */ > static int nodemgr_host_thread(void *__hi) > { > [...] > > for (;;) { > [... sleep until bus reset event ...] > > /* Pause for 1/4 second in 1/16 second intervals, > * to make sure things settle down. */ > g = get_hpsb_generation(host); > for (i = 0; i < 4 ; i++) { > if (msleep_interruptible(63) || > kthread_should_stop()) > goto exit; Totally unrelated, but this looks weird. IMHO you actually wanted: msleep_interruptible(63); if (kthread_should_stop()) goto exit; here, didn't you? Otherwise the thread will exit even when kthread_should_stop() != TRUE (just because it received a signal), and it is not good for a kthread to exit on its own if it uses kthread_should_stop() or if some other piece of kernel code could eventually call kthread_stop(tsk) on it. Ok, probably the thread will never receive a signal in the first place because it's spawned off kthreadd which ignores all signals beforehand, but still ... [PATCH] ieee1394: Fix kthread stopping in nodemgr_host_thread The nodemgr host thread can exit on its own even when kthread_should_stop is not true, on receiving a signal (might never happen in practice, as it ignores signals). But considering kthread_stop() must not be mixed with kthreads that can exit on their own, I think changing the code like this is clearer. This change means the thread can cut its sleep short when receive a signal but looking at the code around, that sounds okay (and again, it might never actually recieve a signal in practice). Signed-off-by: Satyam Sharma <satyam@infradead.org> --- drivers/ieee1394/nodemgr.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/drivers/ieee1394/nodemgr.c b/drivers/ieee1394/nodemgr.c index 2ffd534..981a7da 100644 --- a/drivers/ieee1394/nodemgr.c +++ b/drivers/ieee1394/nodemgr.c @@ -1721,7 +1721,8 @@ static int nodemgr_host_thread(void *__hi) * to make sure things settle down. */ g = get_hpsb_generation(host); for (i = 0; i < 4 ; i++) { - if (msleep_interruptible(63) || kthread_should_stop()) + msleep_interruptible(63); + if (kthread_should_stop()) goto exit; /* Now get the generation in which the node ID's we collect ^ permalink raw reply related [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 14:35 ` Satyam Sharma @ 2007-08-15 14:52 ` Herbert Xu 2007-08-15 16:09 ` Stefan Richter 2007-08-15 19:58 ` Stefan Richter 1 sibling, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-15 14:52 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, Heiko Carstens, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote: > > > I don't know if this here is affected: > > Yes, I think it is. You're clearly expecting the read to actually happen > when you call get_hpsb_generation(). It's clearly not a busy-loop, so > cpu_relax() sounds pointless / wrong solution for this case, so I'm now > somewhat beginning to appreciate the motivation behind this series :-) Nope, we're calling schedule which is a rather heavy-weight barrier. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 14:52 ` Herbert Xu @ 2007-08-15 16:09 ` Stefan Richter 2007-08-15 16:27 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Stefan Richter @ 2007-08-15 16:09 UTC (permalink / raw) To: Herbert Xu Cc: Satyam Sharma, Heiko Carstens, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote: >>> I don't know if this here is affected: [...something like] b = atomic_read(a); for (i = 0; i < 4; i++) { msleep_interruptible(63); c = atomic_read(a); if (c != b) { b = c; i = 0; } } > Nope, we're calling schedule which is a rather heavy-weight > barrier. How does the compiler know that msleep() has got barrier()s? -- Stefan Richter -=====-=-=== =--- -==== http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 16:09 ` Stefan Richter @ 2007-08-15 16:27 ` Paul E. McKenney 2007-08-15 17:13 ` Satyam Sharma 2007-08-15 18:31 ` Segher Boessenkool 0 siblings, 2 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 16:27 UTC (permalink / raw) To: Stefan Richter Cc: Herbert Xu, Satyam Sharma, Heiko Carstens, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 06:09:35PM +0200, Stefan Richter wrote: > Herbert Xu wrote: > > On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote: > >>> I don't know if this here is affected: > > [...something like] > b = atomic_read(a); > for (i = 0; i < 4; i++) { > msleep_interruptible(63); > c = atomic_read(a); > if (c != b) { > b = c; > i = 0; > } > } > > > Nope, we're calling schedule which is a rather heavy-weight > > barrier. > > How does the compiler know that msleep() has got barrier()s? Because msleep_interruptible() is in a separate compilation unit, the compiler has to assume that it might modify any arbitrary global. In many cases, the compiler also has to assume that msleep_interruptible() might call back into a function in the current compilation unit, thus possibly modifying global static variables. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 16:27 ` Paul E. McKenney @ 2007-08-15 17:13 ` Satyam Sharma 2007-08-15 18:31 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 17:13 UTC (permalink / raw) To: Paul E. McKenney Cc: Stefan Richter, Herbert Xu, Heiko Carstens, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, 15 Aug 2007, Paul E. McKenney wrote: > On Wed, Aug 15, 2007 at 06:09:35PM +0200, Stefan Richter wrote: > > Herbert Xu wrote: > > > On Wed, Aug 15, 2007 at 08:05:38PM +0530, Satyam Sharma wrote: > > >>> I don't know if this here is affected: > > > > [...something like] > > b = atomic_read(a); > > for (i = 0; i < 4; i++) { > > msleep_interruptible(63); > > c = atomic_read(a); > > if (c != b) { > > b = c; > > i = 0; > > } > > } > > > > > Nope, we're calling schedule which is a rather heavy-weight > > > barrier. > > > > How does the compiler know that msleep() has got barrier()s? > > Because msleep_interruptible() is in a separate compilation unit, > the compiler has to assume that it might modify any arbitrary global. > In many cases, the compiler also has to assume that msleep_interruptible() > might call back into a function in the current compilation unit, thus > possibly modifying global static variables. Yup, I've just verified this with a testcase. So a call to any function outside of the current compilation unit acts as a compiler barrier. Cool. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 16:27 ` Paul E. McKenney 2007-08-15 17:13 ` Satyam Sharma @ 2007-08-15 18:31 ` Segher Boessenkool 2007-08-15 18:57 ` Paul E. McKenney 1 sibling, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 18:31 UTC (permalink / raw) To: paulmck Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> How does the compiler know that msleep() has got barrier()s? > > Because msleep_interruptible() is in a separate compilation unit, > the compiler has to assume that it might modify any arbitrary global. No; compilation units have nothing to do with it, GCC can optimise across compilation unit boundaries just fine, if you tell it to compile more than one compilation unit at once. What you probably mean is that the compiler has to assume any code it cannot currently see can do anything (insofar as allowed by the relevant standards etc.) > In many cases, the compiler also has to assume that > msleep_interruptible() > might call back into a function in the current compilation unit, thus > possibly modifying global static variables. It most often is smart enough to see what compilation-unit-local variables might be modified that way, though :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 18:31 ` Segher Boessenkool @ 2007-08-15 18:57 ` Paul E. McKenney 2007-08-15 19:54 ` Satyam Sharma 2007-08-15 21:05 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Segher Boessenkool 0 siblings, 2 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 18:57 UTC (permalink / raw) To: Segher Boessenkool Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Wed, Aug 15, 2007 at 08:31:25PM +0200, Segher Boessenkool wrote: > >>How does the compiler know that msleep() has got barrier()s? > > > >Because msleep_interruptible() is in a separate compilation unit, > >the compiler has to assume that it might modify any arbitrary global. > > No; compilation units have nothing to do with it, GCC can optimise > across compilation unit boundaries just fine, if you tell it to > compile more than one compilation unit at once. Last I checked, the Linux kernel build system did compile each .c file as a separate compilation unit. > What you probably mean is that the compiler has to assume any code > it cannot currently see can do anything (insofar as allowed by the > relevant standards etc.) Indeed. > >In many cases, the compiler also has to assume that > >msleep_interruptible() > >might call back into a function in the current compilation unit, thus > >possibly modifying global static variables. > > It most often is smart enough to see what compilation-unit-local > variables might be modified that way, though :-) Yep. For example, if it knows the current value of a given such local variable, and if all code paths that would change some other variable cannot be reached given that current value of the first variable. At least given that gcc doesn't know about multiple threads of execution! Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 18:57 ` Paul E. McKenney @ 2007-08-15 19:54 ` Satyam Sharma 2007-08-15 20:17 ` Paul E. McKenney 2007-08-15 20:47 ` Segher Boessenkool 2007-08-15 21:05 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Segher Boessenkool 1 sibling, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 19:54 UTC (permalink / raw) To: Paul E. McKenney Cc: Segher Boessenkool, horms, Stefan Richter, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang [ The Cc: list scares me. Should probably trim it. ] On Wed, 15 Aug 2007, Paul E. McKenney wrote: > On Wed, Aug 15, 2007 at 08:31:25PM +0200, Segher Boessenkool wrote: > > >>How does the compiler know that msleep() has got barrier()s? > > > > > >Because msleep_interruptible() is in a separate compilation unit, > > >the compiler has to assume that it might modify any arbitrary global. > > > > No; compilation units have nothing to do with it, GCC can optimise > > across compilation unit boundaries just fine, if you tell it to > > compile more than one compilation unit at once. > > Last I checked, the Linux kernel build system did compile each .c file > as a separate compilation unit. > > > What you probably mean is that the compiler has to assume any code > > it cannot currently see can do anything (insofar as allowed by the > > relevant standards etc.) I think this was just terminology confusion here again. Isn't "any code that it cannot currently see" the same as "another compilation unit", and wouldn't the "compilation unit" itself expand if we ask gcc to compile more than one unit at once? Or is there some more specific "definition" for "compilation unit" (in gcc lingo, possibly?) ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 19:54 ` Satyam Sharma @ 2007-08-15 20:17 ` Paul E. McKenney 2007-08-15 20:52 ` Segher Boessenkool 2007-08-15 20:47 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 20:17 UTC (permalink / raw) To: Satyam Sharma Cc: Segher Boessenkool, horms, Stefan Richter, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Thu, Aug 16, 2007 at 01:24:42AM +0530, Satyam Sharma wrote: > [ The Cc: list scares me. Should probably trim it. ] Trim away! ;-) > On Wed, 15 Aug 2007, Paul E. McKenney wrote: > > > On Wed, Aug 15, 2007 at 08:31:25PM +0200, Segher Boessenkool wrote: > > > >>How does the compiler know that msleep() has got barrier()s? > > > > > > > >Because msleep_interruptible() is in a separate compilation unit, > > > >the compiler has to assume that it might modify any arbitrary global. > > > > > > No; compilation units have nothing to do with it, GCC can optimise > > > across compilation unit boundaries just fine, if you tell it to > > > compile more than one compilation unit at once. > > > > Last I checked, the Linux kernel build system did compile each .c file > > as a separate compilation unit. > > > > > What you probably mean is that the compiler has to assume any code > > > it cannot currently see can do anything (insofar as allowed by the > > > relevant standards etc.) > > I think this was just terminology confusion here again. Isn't "any code > that it cannot currently see" the same as "another compilation unit", > and wouldn't the "compilation unit" itself expand if we ask gcc to > compile more than one unit at once? Or is there some more specific > "definition" for "compilation unit" (in gcc lingo, possibly?) This is indeed my understanding -- "compilation unit" is whatever the compiler looks at in one go. I have heard the word "module" used for the minimal compilation unit covering a single .c file and everything that it #includes, but there might be a better name for this. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 20:17 ` Paul E. McKenney @ 2007-08-15 20:52 ` Segher Boessenkool 2007-08-15 22:42 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 20:52 UTC (permalink / raw) To: paulmck Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> I think this was just terminology confusion here again. Isn't "any >> code >> that it cannot currently see" the same as "another compilation unit", >> and wouldn't the "compilation unit" itself expand if we ask gcc to >> compile more than one unit at once? Or is there some more specific >> "definition" for "compilation unit" (in gcc lingo, possibly?) > > This is indeed my understanding -- "compilation unit" is whatever the > compiler looks at in one go. I have heard the word "module" used for > the minimal compilation unit covering a single .c file and everything > that it #includes, but there might be a better name for this. Yes, that's what's called "compilation unit" :-) [/me double checks] Erm, the C standard actually calls it "translation unit". To be exact, to avoid any more confusion: 5.1.1.1/1: A C program need not all be translated at the same time. The text of the program is kept in units called source files, (or preprocessing files) in this International Standard. A source file together with all the headers and source files included via the preprocessing directive #include is known as a preprocessing translation unit. After preprocessing, a preprocessing translation unit is called a translation unit. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 20:52 ` Segher Boessenkool @ 2007-08-15 22:42 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 22:42 UTC (permalink / raw) To: Segher Boessenkool Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Wed, Aug 15, 2007 at 10:52:53PM +0200, Segher Boessenkool wrote: > >>I think this was just terminology confusion here again. Isn't "any > >>code > >>that it cannot currently see" the same as "another compilation unit", > >>and wouldn't the "compilation unit" itself expand if we ask gcc to > >>compile more than one unit at once? Or is there some more specific > >>"definition" for "compilation unit" (in gcc lingo, possibly?) > > > >This is indeed my understanding -- "compilation unit" is whatever the > >compiler looks at in one go. I have heard the word "module" used for > >the minimal compilation unit covering a single .c file and everything > >that it #includes, but there might be a better name for this. > > Yes, that's what's called "compilation unit" :-) > > [/me double checks] > > Erm, the C standard actually calls it "translation unit". > > To be exact, to avoid any more confusion: > > 5.1.1.1/1: > A C program need not all be translated at the same time. The > text of the program is kept in units called source files, (or > preprocessing files) in this International Standard. A source > file together with all the headers and source files included > via the preprocessing directive #include is known as a > preprocessing translation unit. After preprocessing, a > preprocessing translation unit is called a translation unit. I am OK with "translation" and "compilation" being near-synonyms. ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 19:54 ` Satyam Sharma 2007-08-15 20:17 ` Paul E. McKenney @ 2007-08-15 20:47 ` Segher Boessenkool 2007-08-16 0:36 ` (unknown) Satyam Sharma 1 sibling, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 20:47 UTC (permalink / raw) To: Satyam Sharma Cc: horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >>> What you probably mean is that the compiler has to assume any code >>> it cannot currently see can do anything (insofar as allowed by the >>> relevant standards etc.) > > I think this was just terminology confusion here again. Isn't "any code > that it cannot currently see" the same as "another compilation unit", It is not; try gcc -combine or the upcoming link-time optimisation stuff, for example. > and wouldn't the "compilation unit" itself expand if we ask gcc to > compile more than one unit at once? Or is there some more specific > "definition" for "compilation unit" (in gcc lingo, possibly?) "compilation unit" is a C standard term. It typically boils down to "single .c file". Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* (no subject) 2007-08-15 20:47 ` Segher Boessenkool @ 2007-08-16 0:36 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 0:36 UTC (permalink / raw) To: Segher Boessenkool Cc: horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Wed, 15 Aug 2007, Segher Boessenkool wrote: > > > > What you probably mean is that the compiler has to assume any code > > > > it cannot currently see can do anything (insofar as allowed by the > > > > relevant standards etc.) > > > > I think this was just terminology confusion here again. Isn't "any code > > that it cannot currently see" the same as "another compilation unit", > > It is not; try gcc -combine or the upcoming link-time optimisation > stuff, for example. > > > and wouldn't the "compilation unit" itself expand if we ask gcc to > > compile more than one unit at once? Or is there some more specific > > "definition" for "compilation unit" (in gcc lingo, possibly?) > > "compilation unit" is a C standard term. It typically boils down > to "single .c file". As you mentioned later, "single .c file with all the other files (headers or other .c files) that it pulls in via #include" is actually "translation unit", both in the C standard as well as gcc docs. "Compilation unit" doesn't seem to be nearly as standard a term, though in most places it is indeed meant to be same as "translation unit", but with the new gcc inter-module-analysis stuff that you referred to above, I suspect one may reasonably want to call a "compilation unit" as all that the compiler sees at a given instant. BTW I did some auditing (only inside include/asm-{i386,x86_64}/ and arch/{i386,x86_64}/) and found a couple more callsites that don't use cpu_relax(): arch/i386/kernel/crash.c:101 arch/x86_64/kernel/crash.c:97 that are: while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { mdelay(1); msecs--; } where mdelay() becomes __const_udelay() which happens to be in another translation unit (arch/i386/lib/delay.c) and hence saves this callsite from being a bug :-) Curiously, __const_udelay() is still marked as "inline" where it is implemented in lib/delay.c which is weird, considering it won't ever be inlined, would it? With the kernel presently being compiled one translation unit at a time, I don't see how the implementation would be visible to any callsite out there to be able to inline it. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* (unknown) @ 2007-08-16 0:36 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 0:36 UTC (permalink / raw) To: Segher Boessenkool Cc: horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Wed, 15 Aug 2007, Segher Boessenkool wrote: > > > > What you probably mean is that the compiler has to assume any code > > > > it cannot currently see can do anything (insofar as allowed by the > > > > relevant standards etc.) > > > > I think this was just terminology confusion here again. Isn't "any code > > that it cannot currently see" the same as "another compilation unit", > > It is not; try gcc -combine or the upcoming link-time optimisation > stuff, for example. > > > and wouldn't the "compilation unit" itself expand if we ask gcc to > > compile more than one unit at once? Or is there some more specific > > "definition" for "compilation unit" (in gcc lingo, possibly?) > > "compilation unit" is a C standard term. It typically boils down > to "single .c file". As you mentioned later, "single .c file with all the other files (headers or other .c files) that it pulls in via #include" is actually "translation unit", both in the C standard as well as gcc docs. "Compilation unit" doesn't seem to be nearly as standard a term, though in most places it is indeed meant to be same as "translation unit", but with the new gcc inter-module-analysis stuff that you referred to above, I suspect one may reasonably want to call a "compilation unit" as all that the compiler sees at a given instant. BTW I did some auditing (only inside include/asm-{i386,x86_64}/ and arch/{i386,x86_64}/) and found a couple more callsites that don't use cpu_relax(): arch/i386/kernel/crash.c:101 arch/x86_64/kernel/crash.c:97 that are: while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { mdelay(1); msecs--; } where mdelay() becomes __const_udelay() which happens to be in another translation unit (arch/i386/lib/delay.c) and hence saves this callsite from being a bug :-) Curiously, __const_udelay() is still marked as "inline" where it is implemented in lib/delay.c which is weird, considering it won't ever be inlined, would it? With the kernel presently being compiled one translation unit at a time, I don't see how the implementation would be visible to any callsite out there to be able to inline it. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: your mail 2007-08-16 0:36 ` (unknown) Satyam Sharma (?) @ 2007-08-16 0:32 ` Herbert Xu 2007-08-16 0:58 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Satyam Sharma -1 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 0:32 UTC (permalink / raw) To: Satyam Sharma Cc: Segher Boessenkool, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, davem, Linus Torvalds, wensong, wjiang On Thu, Aug 16, 2007 at 06:06:00AM +0530, Satyam Sharma wrote: > > that are: > > while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { > mdelay(1); > msecs--; > } > > where mdelay() becomes __const_udelay() which happens to be in another > translation unit (arch/i386/lib/delay.c) and hence saves this callsite > from being a bug :-) The udelay itself certainly should have some form of cpu_relax in it. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:32 ` your mail Herbert Xu @ 2007-08-16 0:58 ` Satyam Sharma 2007-08-16 0:51 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 0:58 UTC (permalink / raw) To: Herbert Xu Cc: Segher Boessenkool, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, davem, Linus Torvalds, wensong, wjiang [ Sorry for empty subject line in previous mail. I intended to make a patch so cleared it to change it, but ultimately neither made a patch nor restored subject line. Done that now. ] On Thu, 16 Aug 2007, Herbert Xu wrote: > On Thu, Aug 16, 2007 at 06:06:00AM +0530, Satyam Sharma wrote: > > > > that are: > > > > while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { > > mdelay(1); > > msecs--; > > } > > > > where mdelay() becomes __const_udelay() which happens to be in another > > translation unit (arch/i386/lib/delay.c) and hence saves this callsite > > from being a bug :-) > > The udelay itself certainly should have some form of cpu_relax in it. Yes, a form of barrier() must be present in mdelay() or udelay() itself as you say, having it in __const_udelay() is *not* enough (superflous actually, considering it is already a separate translation unit and invisible to the compiler). However, there are no compiler barriers on the macro-definition-path between mdelay(1) and __const_udelay(), so the only thing that saves us from being a bug here is indeed the different-translation-unit concept. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:58 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Satyam Sharma @ 2007-08-16 0:51 ` Herbert Xu 2007-08-16 1:18 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 0:51 UTC (permalink / raw) To: Satyam Sharma Cc: Segher Boessenkool, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, davem, Linus Torvalds, wensong, wjiang On Thu, Aug 16, 2007 at 06:28:42AM +0530, Satyam Sharma wrote: > > > The udelay itself certainly should have some form of cpu_relax in it. > > Yes, a form of barrier() must be present in mdelay() or udelay() itself > as you say, having it in __const_udelay() is *not* enough (superflous > actually, considering it is already a separate translation unit and > invisible to the compiler). As long as __const_udelay does something which has the same effect as barrier it is enough even if it's in the same unit. As a matter of fact it does on i386 where __delay either uses rep_nop or asm/volatile. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:51 ` Herbert Xu @ 2007-08-16 1:18 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 1:18 UTC (permalink / raw) To: Herbert Xu Cc: Segher Boessenkool, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, davem, Linus Torvalds, wensong, wjiang Hi Herbert, On Thu, 16 Aug 2007, Herbert Xu wrote: > On Thu, Aug 16, 2007 at 06:28:42AM +0530, Satyam Sharma wrote: > > > > > The udelay itself certainly should have some form of cpu_relax in it. > > > > Yes, a form of barrier() must be present in mdelay() or udelay() itself > > as you say, having it in __const_udelay() is *not* enough (superflous > > actually, considering it is already a separate translation unit and > > invisible to the compiler). > > As long as __const_udelay does something which has the same > effect as barrier it is enough even if it's in the same unit. Only if __const_udelay() is inlined. But as I said, __const_udelay() -- although marked "inline" -- will never be inlined anywhere in the kernel in reality. It's an exported symbol, and never inlined from modules. Even from built-in targets, the definition of __const_udelay is invisible when gcc is compiling the compilation units of those callsites. The compiler has no idea that that function has barriers or not, so we're saved here _only_ by the lucky fact that __const_udelay() is in a different compilation unit. > As a matter of fact it does on i386 where __delay either uses > rep_nop or asm/volatile. __delay() can be either delay_tsc() or delay_loop() on i386. delay_tsc() uses the rep_nop() there for it's own little busy loop, actually. But for a call site that inlines __const_udelay() -- if it were ever moved to a .h file and marked inline -- the call to __delay() will _still_ be across compilation units. So, again for this case, it does not matter if the callee function has compiler barriers or not (it would've been a different story if we were discussing real/CPU barriers, I think), what saves us here is just the fact that a call is made to a function from a different compilation unit, which is invisible to the compiler when compiling the callsite, and hence acting as the compiler barrier. Regarding delay_loop(), it uses "volatile" for the "asm" which has quite different semantics from the C language "volatile" type-qualifier keyword and does not imply any compiler barrier at all. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: 2007-08-16 0:36 ` (unknown) Satyam Sharma (?) (?) @ 2007-08-16 1:38 ` Segher Boessenkool -1 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 1:38 UTC (permalink / raw) To: Satyam Sharma Cc: horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, ak, netdev, cfriesen, Heiko Carstens, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> "compilation unit" is a C standard term. It typically boils down >> to "single .c file". > > As you mentioned later, "single .c file with all the other files > (headers > or other .c files) that it pulls in via #include" is actually > "translation > unit", both in the C standard as well as gcc docs. Yeah. "single .c file after preprocessing". Same thing :-) > "Compilation unit" > doesn't seem to be nearly as standard a term, though in most places it > is indeed meant to be same as "translation unit", but with the new gcc > inter-module-analysis stuff that you referred to above, I suspect one > may > reasonably want to call a "compilation unit" as all that the compiler > sees > at a given instant. That would be a bit confusing, would it not? They'd better find some better name for that if they want to name it at all (remember, none of these optimisations should have any effect on the semantics of the program, you just get fewer .o files etc.). Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 18:57 ` Paul E. McKenney 2007-08-15 19:54 ` Satyam Sharma @ 2007-08-15 21:05 ` Segher Boessenkool 2007-08-15 22:44 ` Paul E. McKenney 1 sibling, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 21:05 UTC (permalink / raw) To: paulmck Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> No; compilation units have nothing to do with it, GCC can optimise >> across compilation unit boundaries just fine, if you tell it to >> compile more than one compilation unit at once. > > Last I checked, the Linux kernel build system did compile each .c file > as a separate compilation unit. I have some patches to use -combine -fwhole-program for Linux. Highly experimental, you need a patched bleeding edge toolchain. If there's interest I'll clean it up and put it online. David Woodhouse had some similar patches about a year ago. >>> In many cases, the compiler also has to assume that >>> msleep_interruptible() >>> might call back into a function in the current compilation unit, thus >>> possibly modifying global static variables. >> >> It most often is smart enough to see what compilation-unit-local >> variables might be modified that way, though :-) > > Yep. For example, if it knows the current value of a given such local > variable, and if all code paths that would change some other variable > cannot be reached given that current value of the first variable. Or the most common thing: if neither the address of the translation- unit local variable nor the address of any function writing to that variable can "escape" from that translation unit, nothing outside the translation unit can write to the variable. > At least given that gcc doesn't know about multiple threads of > execution! Heh, only about the threads it creates itself (not relevant to the kernel, for sure :-) ) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 21:05 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Segher Boessenkool @ 2007-08-15 22:44 ` Paul E. McKenney 2007-08-16 1:23 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 22:44 UTC (permalink / raw) To: Segher Boessenkool Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Wed, Aug 15, 2007 at 11:05:35PM +0200, Segher Boessenkool wrote: > >>No; compilation units have nothing to do with it, GCC can optimise > >>across compilation unit boundaries just fine, if you tell it to > >>compile more than one compilation unit at once. > > > >Last I checked, the Linux kernel build system did compile each .c file > >as a separate compilation unit. > > I have some patches to use -combine -fwhole-program for Linux. > Highly experimental, you need a patched bleeding edge toolchain. > If there's interest I'll clean it up and put it online. > > David Woodhouse had some similar patches about a year ago. Sounds exciting... ;-) > >>>In many cases, the compiler also has to assume that > >>>msleep_interruptible() > >>>might call back into a function in the current compilation unit, thus > >>>possibly modifying global static variables. > >> > >>It most often is smart enough to see what compilation-unit-local > >>variables might be modified that way, though :-) > > > >Yep. For example, if it knows the current value of a given such local > >variable, and if all code paths that would change some other variable > >cannot be reached given that current value of the first variable. > > Or the most common thing: if neither the address of the translation- > unit local variable nor the address of any function writing to that > variable can "escape" from that translation unit, nothing outside > the translation unit can write to the variable. But there is usually at least one externally callable function in a .c file. > >At least given that gcc doesn't know about multiple threads of > >execution! > > Heh, only about the threads it creates itself (not relevant to > the kernel, for sure :-) ) ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 22:44 ` Paul E. McKenney @ 2007-08-16 1:23 ` Segher Boessenkool 2007-08-16 2:22 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 1:23 UTC (permalink / raw) To: paulmck Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >>>> No; compilation units have nothing to do with it, GCC can optimise >>>> across compilation unit boundaries just fine, if you tell it to >>>> compile more than one compilation unit at once. >>> >>> Last I checked, the Linux kernel build system did compile each .c >>> file >>> as a separate compilation unit. >> >> I have some patches to use -combine -fwhole-program for Linux. >> Highly experimental, you need a patched bleeding edge toolchain. >> If there's interest I'll clean it up and put it online. >> >> David Woodhouse had some similar patches about a year ago. > > Sounds exciting... ;-) Yeah, the breakage is *quite* spectacular :-) >>>>> In many cases, the compiler also has to assume that >>>>> msleep_interruptible() >>>>> might call back into a function in the current compilation unit, >>>>> thus >>>>> possibly modifying global static variables. >>>> >>>> It most often is smart enough to see what compilation-unit-local >>>> variables might be modified that way, though :-) >>> >>> Yep. For example, if it knows the current value of a given such >>> local >>> variable, and if all code paths that would change some other variable >>> cannot be reached given that current value of the first variable. >> >> Or the most common thing: if neither the address of the translation- >> unit local variable nor the address of any function writing to that >> variable can "escape" from that translation unit, nothing outside >> the translation unit can write to the variable. > > But there is usually at least one externally callable function in > a .c file. Of course, but often none of those will (indirectly) write a certain static variable. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:23 ` Segher Boessenkool @ 2007-08-16 2:22 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 2:22 UTC (permalink / raw) To: Segher Boessenkool Cc: horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, Heiko Carstens, jesper.juhl, linux-arch, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Thu, Aug 16, 2007 at 03:23:28AM +0200, Segher Boessenkool wrote: > >>>>No; compilation units have nothing to do with it, GCC can optimise > >>>>across compilation unit boundaries just fine, if you tell it to > >>>>compile more than one compilation unit at once. > >>> > >>>Last I checked, the Linux kernel build system did compile each .c > >>>file > >>>as a separate compilation unit. > >> > >>I have some patches to use -combine -fwhole-program for Linux. > >>Highly experimental, you need a patched bleeding edge toolchain. > >>If there's interest I'll clean it up and put it online. > >> > >>David Woodhouse had some similar patches about a year ago. > > > >Sounds exciting... ;-) > > Yeah, the breakage is *quite* spectacular :-) I bet!!! ;-) > >>>>>In many cases, the compiler also has to assume that > >>>>>msleep_interruptible() > >>>>>might call back into a function in the current compilation unit, > >>>>>thus > >>>>>possibly modifying global static variables. > >>>> > >>>>It most often is smart enough to see what compilation-unit-local > >>>>variables might be modified that way, though :-) > >>> > >>>Yep. For example, if it knows the current value of a given such > >>>local > >>>variable, and if all code paths that would change some other variable > >>>cannot be reached given that current value of the first variable. > >> > >>Or the most common thing: if neither the address of the translation- > >>unit local variable nor the address of any function writing to that > >>variable can "escape" from that translation unit, nothing outside > >>the translation unit can write to the variable. > > > >But there is usually at least one externally callable function in > >a .c file. > > Of course, but often none of those will (indirectly) write a certain > static variable. But there has to be some path to the static functions, assuming that they are not dead code. Yes, there can be cases where the compiler knows enough about the state of the variables to rule out some of code paths to them, but I have no idea how often this happens in kernel code. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 14:35 ` Satyam Sharma 2007-08-15 14:52 ` Herbert Xu @ 2007-08-15 19:58 ` Stefan Richter 1 sibling, 0 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-15 19:58 UTC (permalink / raw) To: Satyam Sharma; +Cc: Linux Kernel Mailing List, Linus Torvalds (trimmed Cc) Satyam Sharma wrote: > [PATCH] ieee1394: Fix kthread stopping in nodemgr_host_thread > > The nodemgr host thread can exit on its own even when kthread_should_stop > is not true, on receiving a signal (might never happen in practice, as > it ignores signals). But considering kthread_stop() must not be mixed with > kthreads that can exit on their own, I think changing the code like this > is clearer. This change means the thread can cut its sleep short when > receive a signal but looking at the code around, that sounds okay (and > again, it might never actually recieve a signal in practice). Thanks, committed to linux1394-2.6.git. -- Stefan Richter -=====-=-=== =--- -==== http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-15 8:18 ` Heiko Carstens 2007-08-15 13:53 ` Stefan Richter @ 2007-08-16 0:39 ` Satyam Sharma 2007-08-24 11:59 ` Denys Vlasenko 1 sibling, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 0:39 UTC (permalink / raw) To: Heiko Carstens Cc: Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, 15 Aug 2007, Heiko Carstens wrote: > [...] > Btw.: we still have > > include/asm-i386/mach-es7000/mach_wakecpu.h: while (!atomic_read(deassert)); > include/asm-i386/mach-default/mach_wakecpu.h: while (!atomic_read(deassert)); > > Looks like they need to be fixed as well. [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() Use cpu_relax() in the busy loops, as atomic_read() doesn't automatically imply volatility for i386 and x86_64. x86_64 doesn't have this issue because it open-codes the while loop in smpboot.c:smp_callin() itself that already uses cpu_relax(). For i386, however, smpboot.c:smp_callin() calls wait_for_init_deassert() which is buggy for mach-default and mach-es7000 cases. [ I test-built a kernel -- smp_callin() itself got inlined in its only callsite, smpboot.c:start_secondary() -- and the relevant piece of code disassembles to the following: 0xc1019704 <start_secondary+12>: mov 0xc144c4c8,%eax 0xc1019709 <start_secondary+17>: test %eax,%eax 0xc101970b <start_secondary+19>: je 0xc1019709 <start_secondary+17> init_deasserted (at 0xc144c4c8) gets fetched into %eax only once and then we loop over the test of the stale value in the register only, so these look like real bugs to me. With the fix below, this becomes: 0xc1019706 <start_secondary+14>: pause 0xc1019708 <start_secondary+16>: cmpl $0x0,0xc144c4c8 0xc101970f <start_secondary+23>: je 0xc1019706 <start_secondary+14> which looks nice and healthy. ] Thanks to Heiko Carstens for noticing this. Signed-off-by: Satyam Sharma <satyam@infradead.org> --- include/asm-i386/mach-default/mach_wakecpu.h | 3 ++- include/asm-i386/mach-es7000/mach_wakecpu.h | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/asm-i386/mach-default/mach_wakecpu.h b/include/asm-i386/mach-default/mach_wakecpu.h index 673b85c..3ebb178 100644 --- a/include/asm-i386/mach-default/mach_wakecpu.h +++ b/include/asm-i386/mach-default/mach_wakecpu.h @@ -15,7 +15,8 @@ static inline void wait_for_init_deassert(atomic_t *deassert) { - while (!atomic_read(deassert)); + while (!atomic_read(deassert)) + cpu_relax(); return; } diff --git a/include/asm-i386/mach-es7000/mach_wakecpu.h b/include/asm-i386/mach-es7000/mach_wakecpu.h index efc903b..84ff583 100644 --- a/include/asm-i386/mach-es7000/mach_wakecpu.h +++ b/include/asm-i386/mach-es7000/mach_wakecpu.h @@ -31,7 +31,8 @@ wakeup_secondary_cpu(int phys_apicid, unsigned long start_eip) static inline void wait_for_init_deassert(atomic_t *deassert) { #ifdef WAKE_SECONDARY_VIA_INIT - while (!atomic_read(deassert)); + while (!atomic_read(deassert)) + cpu_relax(); #endif return; } ^ permalink raw reply related [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-16 0:39 ` [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() Satyam Sharma @ 2007-08-24 11:59 ` Denys Vlasenko 2007-08-24 12:07 ` Andi Kleen ` (3 more replies) 0 siblings, 4 replies; 333+ messages in thread From: Denys Vlasenko @ 2007-08-24 11:59 UTC (permalink / raw) To: Satyam Sharma Cc: Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thursday 16 August 2007 01:39, Satyam Sharma wrote: > > static inline void wait_for_init_deassert(atomic_t *deassert) > { > - while (!atomic_read(deassert)); > + while (!atomic_read(deassert)) > + cpu_relax(); > return; > } For less-than-briliant people like me, it's totally non-obvious that cpu_relax() is needed for correctness here, not just to make P4 happy. IOW: "atomic_read" name quite unambiguously means "I will read this variable from main memory". Which is not true and creates potential for confusion and bugs. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 11:59 ` Denys Vlasenko @ 2007-08-24 12:07 ` Andi Kleen 2007-08-24 12:12 ` Kenn Humborg ` (2 subsequent siblings) 3 siblings, 0 replies; 333+ messages in thread From: Andi Kleen @ 2007-08-24 12:07 UTC (permalink / raw) To: Denys Vlasenko Cc: Satyam Sharma, Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Friday 24 August 2007 13:59:32 Denys Vlasenko wrote: > On Thursday 16 August 2007 01:39, Satyam Sharma wrote: > > > > static inline void wait_for_init_deassert(atomic_t *deassert) > > { > > - while (!atomic_read(deassert)); > > + while (!atomic_read(deassert)) > > + cpu_relax(); > > return; > > } > > For less-than-briliant people like me, it's totally non-obvious that > cpu_relax() is needed for correctness here, not just to make P4 happy. I find it also non obvious. It would be really better to have a barrier or equivalent (volatile or variable clobber) in the atomic_read() > IOW: "atomic_read" name quite unambiguously means "I will read > this variable from main memory". Which is not true and creates > potential for confusion and bugs. Agreed. -Andi ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 11:59 ` Denys Vlasenko @ 2007-08-24 12:12 ` Kenn Humborg 2007-08-24 12:12 ` Kenn Humborg ` (2 subsequent siblings) 3 siblings, 0 replies; 333+ messages in thread From: Kenn Humborg @ 2007-08-24 12:12 UTC (permalink / raw) To: Denys Vlasenko, Satyam Sharma Cc: Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher > On Thursday 16 August 2007 01:39, Satyam Sharma wrote: > > > > static inline void wait_for_init_deassert(atomic_t *deassert) > > { > > - while (!atomic_read(deassert)); > > + while (!atomic_read(deassert)) > > + cpu_relax(); > > return; > > } > > For less-than-briliant people like me, it's totally non-obvious that > cpu_relax() is needed for correctness here, not just to make P4 happy. > > IOW: "atomic_read" name quite unambiguously means "I will read > this variable from main memory". Which is not true and creates > potential for confusion and bugs. To me, "atomic_read" means a read which is synchronized with other changes to the variable (using the atomic_XXX functions) in such a way that I will always only see the "before" or "after" state of the variable - never an intermediate state while a modification is happening. It doesn't imply that I have to see the "after" state immediately after another thread modifies it. Perhaps the Linux atomic_XXX functions work like that, or used to work like that, but it's counter-intuitive to me that "atomic" should imply a memory read. Later, Kenn ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() @ 2007-08-24 12:12 ` Kenn Humborg 0 siblings, 0 replies; 333+ messages in thread From: Kenn Humborg @ 2007-08-24 12:12 UTC (permalink / raw) To: Denys Vlasenko, Satyam Sharma Cc: Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher > On Thursday 16 August 2007 01:39, Satyam Sharma wrote: > > > > static inline void wait_for_init_deassert(atomic_t *deassert) > > { > > - while (!atomic_read(deassert)); > > + while (!atomic_read(deassert)) > > + cpu_relax(); > > return; > > } > > For less-than-briliant people like me, it's totally non-obvious that > cpu_relax() is needed for correctness here, not just to make P4 happy. > > IOW: "atomic_read" name quite unambiguously means "I will read > this variable from main memory". Which is not true and creates > potential for confusion and bugs. To me, "atomic_read" means a read which is synchronized with other changes to the variable (using the atomic_XXX functions) in such a way that I will always only see the "before" or "after" state of the variable - never an intermediate state while a modification is happening. It doesn't imply that I have to see the "after" state immediately after another thread modifies it. Perhaps the Linux atomic_XXX functions work like that, or used to work like that, but it's counter-intuitive to me that "atomic" should imply a memory read. Later, Kenn ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 12:12 ` Kenn Humborg (?) @ 2007-08-24 14:25 ` Denys Vlasenko 2007-08-24 17:34 ` Linus Torvalds -1 siblings, 1 reply; 333+ messages in thread From: Denys Vlasenko @ 2007-08-24 14:25 UTC (permalink / raw) To: Kenn Humborg Cc: Satyam Sharma, Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Friday 24 August 2007 13:12, Kenn Humborg wrote: > > On Thursday 16 August 2007 01:39, Satyam Sharma wrote: > > > static inline void wait_for_init_deassert(atomic_t *deassert) > > > { > > > - while (!atomic_read(deassert)); > > > + while (!atomic_read(deassert)) > > > + cpu_relax(); > > > return; > > > } > > > > For less-than-briliant people like me, it's totally non-obvious that > > cpu_relax() is needed for correctness here, not just to make P4 happy. > > > > IOW: "atomic_read" name quite unambiguously means "I will read > > this variable from main memory". Which is not true and creates > > potential for confusion and bugs. > > To me, "atomic_read" means a read which is synchronized with other > changes to the variable (using the atomic_XXX functions) in such > a way that I will always only see the "before" or "after" > state of the variable - never an intermediate state while a > modification is happening. It doesn't imply that I have to > see the "after" state immediately after another thread modifies > it. So you are ok with compiler propagating n1 to n2 here: n1 += atomic_read(x); other_variable++; n2 += atomic_read(x); without accessing x second time. What's the point? Any sane coder will say that explicitly anyway: tmp = atomic_read(x); n1 += tmp; other_variable++; n2 += tmp; if only for the sake of code readability. Because first code is definitely hinting that it reads RAM twice, and it's actively *bad* for code readability when in fact it's not the case! Locking, compiler and CPU barriers are complicated enough already, please don't make them even harder to understand. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 14:25 ` Denys Vlasenko @ 2007-08-24 17:34 ` Linus Torvalds 0 siblings, 0 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-24 17:34 UTC (permalink / raw) To: Denys Vlasenko Cc: Kenn Humborg, Satyam Sharma, Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 24 Aug 2007, Denys Vlasenko wrote: > > So you are ok with compiler propagating n1 to n2 here: > > n1 += atomic_read(x); > other_variable++; > n2 += atomic_read(x); > > without accessing x second time. What's the point? Any sane coder > will say that explicitly anyway: No. This is a common mistake, and it's total crap. Any "sane coder" will often use inline functions, macros, etc helpers to do certain abstract things. Those things may contain "atomic_read()" calls. The biggest reason for compilers doing CSE is exactly the fact that many opportunities for CSE simple *are*not*visible* on a source code level. That is true of things like atomic_read() equally as to things like shared offsets inside structure member accesses. No difference what-so-ever. Yes, we have, traditionally, tried to make it *easy* for the compiler to generate good code. So when we can, and when we look at performance for some really hot path, we *will* write the source code so that the compiler doesn't even have the option to screw it up, and that includes things like doing CSE at a source code level so that we don't see the compiler re-doing accesses unnecessarily. And I'm not saying we shouldn't do that. But "performance" is not an either-or kind of situation, and we should: - spend the time at a source code level: make it reasonably easy for the compiler to generate good code, and use the right algorithms at a higher level (and order structures etc so that they have good cache behaviour). - .. *and* expect the compiler to handle the cases we didn't do by hand pretty well anyway. In particular, quite often, abstraction levels at a software level means that we give compilers "stupid" code, because some function may have a certain high-level abstraction rule, but then on a particular architecture it's actually a no-op, and the compiler should get to "untangle" our stupid code and generate good end results. - .. *and* expect the hardware to be sane and do a good job even when the compiler didn't generate perfect code or there were unlucky cache miss patterns etc. and if we do all of that, we'll get good performance. But you really do want all three levels. It's not enough to be good at any one level (or even any two). Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 11:59 ` Denys Vlasenko 2007-08-24 12:07 ` Andi Kleen 2007-08-24 12:12 ` Kenn Humborg @ 2007-08-24 13:30 ` Satyam Sharma 2007-08-24 17:06 ` Christoph Lameter 2007-08-24 16:19 ` Luck, Tony 3 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-24 13:30 UTC (permalink / raw) To: Denys Vlasenko Cc: Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Nick Piggin Hi Denys, On Fri, 24 Aug 2007, Denys Vlasenko wrote: > On Thursday 16 August 2007 01:39, Satyam Sharma wrote: > > > > static inline void wait_for_init_deassert(atomic_t *deassert) > > { > > - while (!atomic_read(deassert)); > > + while (!atomic_read(deassert)) > > + cpu_relax(); > > return; > > } > > For less-than-briliant people like me, it's totally non-obvious that > cpu_relax() is needed for correctness here, not just to make P4 happy. This thread has been round-and-round with exactly the same discussions :-) I had proposed few such variants to make a compiler barrier implicit in atomic_{read,set} myself, but frankly, at least personally speaking (now that I know better), I'm not so much in favour of implicit barriers (compiler, memory or both) in atomic_{read,set}. This might sound like an about-turn if you read my own postings to Nick Piggin from a week back, but I do agree with most his opinions on the matter now -- separation of barriers from atomic ops is actually good, beneficial to certain code that knows what it's doing, explicit usage of barriers stands out more clearly (most people here who deal with it do know cpu_relax() is an explicit compiler barrier) compared to an implicit usage in an atomic_read() or such variant ... > IOW: "atomic_read" name quite unambiguously means "I will read > this variable from main memory". Which is not true and creates > potential for confusion and bugs. I'd have to disagree here -- atomic ops are all about _atomicity_ of memory accesses, not _making_ them happen (or visible to other CPUs) _then and there_ itself. The latter are the job of barriers. The behaviour (and expectations) are quite comprehensively covered in atomic_ops.txt -- let alone atomic_{read,set}, even atomic_{inc,dec} are permitted by archs' implementations to _not_ have any memory barriers, for that matter. [It is unrelated that on x86 making them SMP-safe requires the use of the LOCK prefix that also happens to be an implicit memory barrier.] An argument was also made about consistency of atomic_{read,set} w.r.t. the other atomic ops -- but clearly, they are all already consistent! All of them are atomic :-) The fact that atomic_{read,set} do _not_ require any inline asm or LOCK prefix whereas the others do, has to do with the fact that unlike all others, atomic_{read,set} are not RMW ops and hence guaranteed to be atomic just as they are in plain & simple C. But if people do seem to have a mixed / confused notion of atomicity and barriers, and if there's consensus, then as I'd said earlier, I have no issues in going with the consensus (eg. having API variants). Linus would be more difficult to convince, however, I suspect :-) Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 13:30 ` Satyam Sharma @ 2007-08-24 17:06 ` Christoph Lameter 2007-08-24 20:26 ` Denys Vlasenko 0 siblings, 1 reply; 333+ messages in thread From: Christoph Lameter @ 2007-08-24 17:06 UTC (permalink / raw) To: Satyam Sharma Cc: Denys Vlasenko, Heiko Carstens, Herbert Xu, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Nick Piggin On Fri, 24 Aug 2007, Satyam Sharma wrote: > But if people do seem to have a mixed / confused notion of atomicity > and barriers, and if there's consensus, then as I'd said earlier, I > have no issues in going with the consensus (eg. having API variants). > Linus would be more difficult to convince, however, I suspect :-) The confusion may be the result of us having barrier semantics in atomic_read. If we take that out then we may avoid future confusions. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 17:06 ` Christoph Lameter @ 2007-08-24 20:26 ` Denys Vlasenko 2007-08-24 20:34 ` Chris Snook 0 siblings, 1 reply; 333+ messages in thread From: Denys Vlasenko @ 2007-08-24 20:26 UTC (permalink / raw) To: Christoph Lameter Cc: Satyam Sharma, Heiko Carstens, Herbert Xu, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Nick Piggin On Friday 24 August 2007 18:06, Christoph Lameter wrote: > On Fri, 24 Aug 2007, Satyam Sharma wrote: > > But if people do seem to have a mixed / confused notion of atomicity > > and barriers, and if there's consensus, then as I'd said earlier, I > > have no issues in going with the consensus (eg. having API variants). > > Linus would be more difficult to convince, however, I suspect :-) > > The confusion may be the result of us having barrier semantics in > atomic_read. If we take that out then we may avoid future confusions. I think better name may help. Nuke atomic_read() altogether. n = atomic_value(x); // doesnt hint as strongly at reading as "atomic_read" n = atomic_fetch(x); // yes, we _do_ touch RAM n = atomic_read_uncached(x); // or this How does that sound? -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 20:26 ` Denys Vlasenko @ 2007-08-24 20:34 ` Chris Snook 0 siblings, 0 replies; 333+ messages in thread From: Chris Snook @ 2007-08-24 20:34 UTC (permalink / raw) To: Denys Vlasenko Cc: Christoph Lameter, Satyam Sharma, Heiko Carstens, Herbert Xu, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Nick Piggin Denys Vlasenko wrote: > On Friday 24 August 2007 18:06, Christoph Lameter wrote: >> On Fri, 24 Aug 2007, Satyam Sharma wrote: >>> But if people do seem to have a mixed / confused notion of atomicity >>> and barriers, and if there's consensus, then as I'd said earlier, I >>> have no issues in going with the consensus (eg. having API variants). >>> Linus would be more difficult to convince, however, I suspect :-) >> The confusion may be the result of us having barrier semantics in >> atomic_read. If we take that out then we may avoid future confusions. > > I think better name may help. Nuke atomic_read() altogether. > > n = atomic_value(x); // doesnt hint as strongly at reading as "atomic_read" > n = atomic_fetch(x); // yes, we _do_ touch RAM > n = atomic_read_uncached(x); // or this > > How does that sound? atomic_value() vs. atomic_fetch() should be rather unambiguous. atomic_read_uncached() begs the question of precisely which cache we are avoiding, and could itself cause confusion. So, if I were writing atomic.h from scratch, knowing what I know now, I think I'd use atomic_value() and atomic_fetch(). The problem is that there are a lot of existing users of atomic_read(), and we can't write a script to correctly guess their intent. I'm not sure auditing all uses of atomic_read() is really worth the comparatively miniscule benefits. We could play it safe and convert them all to atomic_fetch(), or we could acknowledge that changing the semantics 8 months ago was not at all disastrous, and make them all atomic_value(), allowing people to use atomic_fetch() where they really care. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() 2007-08-24 11:59 ` Denys Vlasenko @ 2007-08-24 16:19 ` Luck, Tony 2007-08-24 12:12 ` Kenn Humborg ` (2 subsequent siblings) 3 siblings, 0 replies; 333+ messages in thread From: Luck, Tony @ 2007-08-24 16:19 UTC (permalink / raw) To: Denys Vlasenko, Satyam Sharma Cc: Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher >> static inline void wait_for_init_deassert(atomic_t *deassert) >> { >> - while (!atomic_read(deassert)); >> + while (!atomic_read(deassert)) >> + cpu_relax(); >> return; >> } > > For less-than-briliant people like me, it's totally non-obvious that > cpu_relax() is needed for correctness here, not just to make P4 happy. Not just P4 ... there are other threaded cpus where it is useful to let the core know that this is a busy loop so it would be a good thing to let other threads have priority. Even on a non-threaded cpu the cpu_relax() could be useful in the future to hint to the cpu that it could drop into a lower power hogging state. But I agree with your main point that the loop without the cpu_relax() looks like it ought to work because atomic_read() ought to actually go out and read memory each time around the loop. -Tony ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() @ 2007-08-24 16:19 ` Luck, Tony 0 siblings, 0 replies; 333+ messages in thread From: Luck, Tony @ 2007-08-24 16:19 UTC (permalink / raw) To: Denys Vlasenko, Satyam Sharma Cc: Heiko Carstens, Herbert Xu, Chris Snook, clameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher >> static inline void wait_for_init_deassert(atomic_t *deassert) >> { >> - while (!atomic_read(deassert)); >> + while (!atomic_read(deassert)) >> + cpu_relax(); >> return; >> } > > For less-than-briliant people like me, it's totally non-obvious that > cpu_relax() is needed for correctness here, not just to make P4 happy. Not just P4 ... there are other threaded cpus where it is useful to let the core know that this is a busy loop so it would be a good thing to let other threads have priority. Even on a non-threaded cpu the cpu_relax() could be useful in the future to hint to the cpu that it could drop into a lower power hogging state. But I agree with your main point that the loop without the cpu_relax() looks like it ought to work because atomic_read() ought to actually go out and read memory each time around the loop. -Tony ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 6:49 ` Herbert Xu (?) (?) @ 2007-08-15 16:13 ` Chris Snook 2007-08-15 23:40 ` Herbert Xu -1 siblings, 1 reply; 333+ messages in thread From: Chris Snook @ 2007-08-15 16:13 UTC (permalink / raw) To: Herbert Xu Cc: satyam, clameter, linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > Chris Snook <csnook@redhat.com> wrote: >> Because atomic operations are generally used for synchronization, which requires >> volatile behavior. Most such codepaths currently use an inefficient barrier(). >> Some forget to and we get bugs, because people assume that atomic_read() >> actually reads something, and atomic_write() actually writes something. Worse, >> these are architecture-specific, even compiler version-specific bugs that are >> often difficult to track down. > > I'm yet to see a single example from the current tree where > this patch series is the correct solution. So far the only > example has been a buggy piece of code which has since been > fixed with a cpu_relax. Part of the motivation here is to fix heisenbugs. If I knew where they were, I'd be posting patches for them. Unlike most bugs, where we want to expose them as obviously as possible, these can be extremely difficult to track down, and are often due to people assuming that the atomic_* operations have the same semantics they've historically had. Remember that until recently, all SMP architectures except s390 (which very few kernel developers outside of IBM, Red Hat, and SuSE do much work on) had volatile declarations for atomic_t. Removing the volatile declarations from i386 and x86_64 may have created heisenbugs that won't manifest themselves until GCC 6.0 comes out and people start compiling kernels with -O5. We should have consistent semantics for atomic_* operations. The other motivation is to reduce the need for the barriers used to prevent/fix such problems which clobber all your registers, and instead force atomic_* operations to behave in the way they're actually used. After the (resubmitted) patchset is merged, we'll be able to remove a whole bunch of barriers, shrinking our source and our binaries, and improving performance. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 16:13 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Chris Snook @ 2007-08-15 23:40 ` Herbert Xu 2007-08-15 23:51 ` Paul E. McKenney 2007-08-16 1:26 ` Segher Boessenkool 0 siblings, 2 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-15 23:40 UTC (permalink / raw) To: Chris Snook Cc: satyam, clameter, linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 12:13:12PM -0400, Chris Snook wrote: > > Part of the motivation here is to fix heisenbugs. If I knew where they By the same token we should probably disable optimisations altogether since that too can create heisenbugs. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 23:40 ` Herbert Xu @ 2007-08-15 23:51 ` Paul E. McKenney 2007-08-16 1:30 ` Segher Boessenkool 2007-08-16 1:26 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 23:51 UTC (permalink / raw) To: Herbert Xu Cc: Chris Snook, satyam, clameter, linux-kernel, linux-arch, torvalds, netdev, akpm, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 07:40:21AM +0800, Herbert Xu wrote: > On Wed, Aug 15, 2007 at 12:13:12PM -0400, Chris Snook wrote: > > > > Part of the motivation here is to fix heisenbugs. If I knew where they > > By the same token we should probably disable optimisations > altogether since that too can create heisenbugs. Precisely the point -- use of volatile (whether in casts or on asms) in these cases are intended to disable those optimizations likely to result in heisenbugs. But they are also intended to leave other valuable optimizations in force. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 23:51 ` Paul E. McKenney @ 2007-08-16 1:30 ` Segher Boessenkool 2007-08-16 2:30 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 1:30 UTC (permalink / raw) To: paulmck Cc: heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang >>> Part of the motivation here is to fix heisenbugs. If I knew where >>> they >> >> By the same token we should probably disable optimisations >> altogether since that too can create heisenbugs. > > Precisely the point -- use of volatile (whether in casts or on asms) > in these cases are intended to disable those optimizations likely to > result in heisenbugs. The only thing volatile on an asm does is create a side effect on the asm statement; in effect, it tells the compiler "do not remove this asm even if you don't need any of its outputs". It's not disabling optimisation likely to result in bugs, heisen- or otherwise; _not_ putting the volatile on an asm that needs it simply _is_ a bug :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:30 ` Segher Boessenkool @ 2007-08-16 2:30 ` Paul E. McKenney 2007-08-16 19:33 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 2:30 UTC (permalink / raw) To: Segher Boessenkool Cc: heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Thu, Aug 16, 2007 at 03:30:44AM +0200, Segher Boessenkool wrote: > >>>Part of the motivation here is to fix heisenbugs. If I knew where > >>>they > >> > >>By the same token we should probably disable optimisations > >>altogether since that too can create heisenbugs. > > > >Precisely the point -- use of volatile (whether in casts or on asms) > >in these cases are intended to disable those optimizations likely to > >result in heisenbugs. > > The only thing volatile on an asm does is create a side effect > on the asm statement; in effect, it tells the compiler "do not > remove this asm even if you don't need any of its outputs". > > It's not disabling optimisation likely to result in bugs, > heisen- or otherwise; _not_ putting the volatile on an asm > that needs it simply _is_ a bug :-) Yep. And the reason it is a bug is that it fails to disable the relevant compiler optimizations. So I suspect that we might actually be saying the same thing here. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:30 ` Paul E. McKenney @ 2007-08-16 19:33 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 19:33 UTC (permalink / raw) To: paulmck Cc: heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang >> The only thing volatile on an asm does is create a side effect >> on the asm statement; in effect, it tells the compiler "do not >> remove this asm even if you don't need any of its outputs". >> >> It's not disabling optimisation likely to result in bugs, >> heisen- or otherwise; _not_ putting the volatile on an asm >> that needs it simply _is_ a bug :-) > > Yep. And the reason it is a bug is that it fails to disable > the relevant compiler optimizations. So I suspect that we might > actually be saying the same thing here. We're not saying the same thing, but we do agree :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 23:40 ` Herbert Xu 2007-08-15 23:51 ` Paul E. McKenney @ 2007-08-16 1:26 ` Segher Boessenkool 2007-08-16 2:23 ` Nick Piggin 1 sibling, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 1:26 UTC (permalink / raw) To: Herbert Xu Cc: heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, davem, wensong, wjiang >> Part of the motivation here is to fix heisenbugs. If I knew where >> they > > By the same token we should probably disable optimisations > altogether since that too can create heisenbugs. Almost everything is a tradeoff; and so is this. I don't believe most people would find disabling all compiler optimisations an acceptable price to pay for some peace of mind. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:26 ` Segher Boessenkool @ 2007-08-16 2:23 ` Nick Piggin 2007-08-16 19:32 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-16 2:23 UTC (permalink / raw) To: Segher Boessenkool Cc: Herbert Xu, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, davem, wensong, wjiang Segher Boessenkool wrote: >>> Part of the motivation here is to fix heisenbugs. If I knew where they >> >> >> By the same token we should probably disable optimisations >> altogether since that too can create heisenbugs. > > > Almost everything is a tradeoff; and so is this. I don't > believe most people would find disabling all compiler > optimisations an acceptable price to pay for some peace > of mind. So why is this a good tradeoff? I also think that just adding things to APIs in the hope it might fix up some bugs isn't really a good road to go down. Where do you stop? On the actual proposal to make atomic_operators volatile: I think the better approach in the long term, for both maintainability of the code and education of coders, is to make the use of barriers _more_ explicit rather than sprinkling these "just in case" ones around. You may get rid of a few atomic_read heisenbugs (in noise when compared to all bugs), but if the coder was using a regular atomic load, or a test_bit (which is also atomic), etc. then they're going to have problems. It would be better for Linux if everyone was to have better awareness of barriers than to hide some of the cases where they're required. A pretty large number of bugs I see in lock free code in the VM is due to memory ordering problems. It's hard to find those bugs, or even be aware when you're writing buggy code if you don't have some feel for barriers. -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:23 ` Nick Piggin @ 2007-08-16 19:32 ` Segher Boessenkool 2007-08-17 2:19 ` Nick Piggin 0 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 19:32 UTC (permalink / raw) To: Nick Piggin Cc: heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang >>>> Part of the motivation here is to fix heisenbugs. If I knew where >>>> they >>> >>> >>> By the same token we should probably disable optimisations >>> altogether since that too can create heisenbugs. >> Almost everything is a tradeoff; and so is this. I don't >> believe most people would find disabling all compiler >> optimisations an acceptable price to pay for some peace >> of mind. > > So why is this a good tradeoff? It certainly is better than disabling all compiler optimisations! > I also think that just adding things to APIs in the hope it might fix > up some bugs isn't really a good road to go down. Where do you stop? I look at it the other way: keeping the "volatile" semantics in atomic_XXX() (or adding them to it, whatever) helps _prevent_ bugs; certainly most people expect that behaviour, and also that behaviour is *needed* in some places and no other interface provides that functionality. [some confusion about barriers wrt atomics snipped] Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 19:32 ` Segher Boessenkool @ 2007-08-17 2:19 ` Nick Piggin 2007-08-17 3:16 ` Paul Mackerras 2007-08-17 17:37 ` Segher Boessenkool 0 siblings, 2 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 2:19 UTC (permalink / raw) To: Segher Boessenkool Cc: heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Segher Boessenkool wrote: >>>>> Part of the motivation here is to fix heisenbugs. If I knew where >>>>> they >>>> >>>> >>>> >>>> By the same token we should probably disable optimisations >>>> altogether since that too can create heisenbugs. >>> >>> Almost everything is a tradeoff; and so is this. I don't >>> believe most people would find disabling all compiler >>> optimisations an acceptable price to pay for some peace >>> of mind. >> >> >> So why is this a good tradeoff? > > > It certainly is better than disabling all compiler optimisations! It's easy to be better than something really stupid :) So i386 and x86-64 don't have volatiles there, and it saves them a few K of kernel text. What you need to justify is why it is a good tradeoff to make them volatile (which btw, is much harder to go the other way after we let people make those assumptions). >> I also think that just adding things to APIs in the hope it might fix >> up some bugs isn't really a good road to go down. Where do you stop? > > > I look at it the other way: keeping the "volatile" semantics in > atomic_XXX() (or adding them to it, whatever) helps _prevent_ bugs; Yeah, but we could add lots of things to help prevent bugs and would never be included. I would also contend that it helps _hide_ bugs and encourages people to be lazy when thinking about these things. Also, you dismiss the fact that we'd actually be *adding* volatile semantics back to the 2 most widely tested architectures (in terms of test time, number of testers, variety of configurations, and coverage of driver code). This is a very important different from just keeping volatile semantics because it is basically a one-way API change. > certainly most people expect that behaviour, and also that behaviour > is *needed* in some places and no other interface provides that > functionality. I don't know that most people would expect that behaviour. Is there any documentation anywhere that would suggest this? Also, barrier() most definitely provides the required functionality. It is overkill in some situations, but volatile is overkill in _most_ situations. If that's what you're worried about, we should add a new ordering primitive. > [some confusion about barriers wrt atomics snipped] What were you confused about? -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 2:19 ` Nick Piggin @ 2007-08-17 3:16 ` Paul Mackerras 2007-08-17 3:32 ` Nick Piggin 2007-08-17 3:42 ` Linus Torvalds 2007-08-17 17:37 ` Segher Boessenkool 1 sibling, 2 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 3:16 UTC (permalink / raw) To: Nick Piggin Cc: Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Nick Piggin writes: > So i386 and x86-64 don't have volatiles there, and it saves them a > few K of kernel text. What you need to justify is why it is a good I'm really surprised it's as much as a few K. I tried it on powerpc and it only saved 40 bytes (10 instructions) for a G5 config. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:16 ` Paul Mackerras @ 2007-08-17 3:32 ` Nick Piggin 2007-08-17 3:50 ` Linus Torvalds 2007-08-17 3:42 ` Linus Torvalds 1 sibling, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-17 3:32 UTC (permalink / raw) To: Paul Mackerras Cc: Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Paul Mackerras wrote: > Nick Piggin writes: > > >>So i386 and x86-64 don't have volatiles there, and it saves them a >>few K of kernel text. What you need to justify is why it is a good > > > I'm really surprised it's as much as a few K. I tried it on powerpc > and it only saved 40 bytes (10 instructions) for a G5 config. > > Paul. > I'm surprised too. Numbers were from the "...use asm() like the other atomic operations already do" thread. According to them, text data bss dec hex filename 3434150 249176 176128 3859454 3ae3fe atomic_normal/vmlinux 3436203 249176 176128 3861507 3aec03 atomic_volatile/vmlinux The first one is a stock kenel, the second is with atomic_read/set cast to volatile. gcc-4.1 -- maybe if you have an earlier gcc it won't optimise as much? -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:32 ` Nick Piggin @ 2007-08-17 3:50 ` Linus Torvalds 2007-08-17 23:59 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Linus Torvalds @ 2007-08-17 3:50 UTC (permalink / raw) To: Nick Piggin Cc: Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Fri, 17 Aug 2007, Nick Piggin wrote: > > I'm surprised too. Numbers were from the "...use asm() like the other > atomic operations already do" thread. According to them, > > text data bss dec hex filename > 3434150 249176 176128 3859454 3ae3fe atomic_normal/vmlinux > 3436203 249176 176128 3861507 3aec03 atomic_volatile/vmlinux > > The first one is a stock kenel, the second is with atomic_read/set > cast to volatile. gcc-4.1 -- maybe if you have an earlier gcc it > won't optimise as much? No, see my earlier reply. "volatile" really *is* an incredible piece of crap. Just try it yourself: volatile int i; int j; int testme(void) { return i <= 1; } int testme2(void) { return j <= 1; } and compile with all the optimizations you can. I get: testme: movl i(%rip), %eax subl $1, %eax setle %al movzbl %al, %eax ret vs testme2: xorl %eax, %eax cmpl $1, j(%rip) setle %al ret (now, whether that "xorl + setle" is better than "setle + movzbl", I don't really know - maybe it is. But that's not thepoint. The point is the difference between movl i(%rip), %eax subl $1, %eax and cmpl $1, j(%rip) and imagine this being done for *every* single volatile access. Just do a git grep atomic_read to see how atomics are actually used. A lot of them are exactly the above kind of "compare against a value". Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:50 ` Linus Torvalds @ 2007-08-17 23:59 ` Paul E. McKenney 2007-08-18 0:09 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 23:59 UTC (permalink / raw) To: Linus Torvalds Cc: Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Thu, Aug 16, 2007 at 08:50:30PM -0700, Linus Torvalds wrote: > Just try it yourself: > > volatile int i; > int j; > > int testme(void) > { > return i <= 1; > } > > int testme2(void) > { > return j <= 1; > } > > and compile with all the optimizations you can. > > I get: > > testme: > movl i(%rip), %eax > subl $1, %eax > setle %al > movzbl %al, %eax > ret > > vs > > testme2: > xorl %eax, %eax > cmpl $1, j(%rip) > setle %al > ret > > (now, whether that "xorl + setle" is better than "setle + movzbl", I don't > really know - maybe it is. But that's not thepoint. The point is the > difference between > > movl i(%rip), %eax > subl $1, %eax > > and > > cmpl $1, j(%rip) gcc bugzilla bug #33102, for whatever that ends up being worth. ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 23:59 ` Paul E. McKenney @ 2007-08-18 0:09 ` Herbert Xu 2007-08-18 1:08 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-18 0:09 UTC (permalink / raw) To: Paul E. McKenney Cc: Linus Torvalds, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, davem, wensong, wjiang On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote: > > gcc bugzilla bug #33102, for whatever that ends up being worth. ;-) I had totally forgotten that I'd already filed that bug more than six years ago until they just closed yours as a duplicate of mine :) Good luck in getting it fixed! Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 0:09 ` Herbert Xu @ 2007-08-18 1:08 ` Paul E. McKenney 2007-08-18 1:24 ` Christoph Lameter 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-18 1:08 UTC (permalink / raw) To: Herbert Xu Cc: Linus Torvalds, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, davem, wensong, wjiang On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote: > On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote: > > > > gcc bugzilla bug #33102, for whatever that ends up being worth. ;-) > > I had totally forgotten that I'd already filed that bug more > than six years ago until they just closed yours as a duplicate > of mine :) > > Good luck in getting it fixed! Well, just got done re-opening it for the third time. And a local gcc community member advised me not to give up too easily. But I must admit that I am impressed with the speed that it was identified as duplicate. Should be entertaining! ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 1:08 ` Paul E. McKenney @ 2007-08-18 1:24 ` Christoph Lameter 2007-08-18 1:41 ` Satyam Sharma ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-18 1:24 UTC (permalink / raw) To: Paul E. McKenney Cc: Herbert Xu, Linus Torvalds, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, schwidefsky, Chris Snook, davem, wensong, wjiang On Fri, 17 Aug 2007, Paul E. McKenney wrote: > On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote: > > On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote: > > > > > > gcc bugzilla bug #33102, for whatever that ends up being worth. ;-) > > > > I had totally forgotten that I'd already filed that bug more > > than six years ago until they just closed yours as a duplicate > > of mine :) > > > > Good luck in getting it fixed! > > Well, just got done re-opening it for the third time. And a local > gcc community member advised me not to give up too easily. But I > must admit that I am impressed with the speed that it was identified > as duplicate. > > Should be entertaining! ;-) Right. ROTFL... volatile actually breaks atomic_t instead of making it safe. x++ becomes a register load, increment and a register store. Without volatile we can increment the memory directly. It seems that volatile requires that the variable is loaded into a register first and then operated upon. Understandable when you think about volatile being used to access memory mapped I/O registers where a RMW operation could be problematic. See http://gcc.gnu.org/bugzilla/show_bug.cgi?id=3506 ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 1:24 ` Christoph Lameter @ 2007-08-18 1:41 ` Satyam Sharma 2007-08-18 4:13 ` Linus Torvalds 2007-08-18 21:56 ` Paul E. McKenney 2007-08-20 13:31 ` Chris Snook 2 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-18 1:41 UTC (permalink / raw) To: Christoph Lameter Cc: Paul E. McKenney, Herbert Xu, Linus Torvalds, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Fri, 17 Aug 2007, Christoph Lameter wrote: > On Fri, 17 Aug 2007, Paul E. McKenney wrote: > > > On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote: > > > On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote: > > > > > > > > gcc bugzilla bug #33102, for whatever that ends up being worth. ;-) > > > > > > I had totally forgotten that I'd already filed that bug more > > > than six years ago until they just closed yours as a duplicate > > > of mine :) > > > > > > Good luck in getting it fixed! > > > > Well, just got done re-opening it for the third time. And a local > > gcc community member advised me not to give up too easily. But I > > must admit that I am impressed with the speed that it was identified > > as duplicate. > > > > Should be entertaining! ;-) > > Right. ROTFL... volatile actually breaks atomic_t instead of making it > safe. x++ becomes a register load, increment and a register store. Without > volatile we can increment the memory directly. No code does (or would do, or should do): x.counter++; on an "atomic_t x;" anyway. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 1:41 ` Satyam Sharma @ 2007-08-18 4:13 ` Linus Torvalds 2007-08-18 13:36 ` Satyam Sharma ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-18 4:13 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul E. McKenney, Herbert Xu, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Sat, 18 Aug 2007, Satyam Sharma wrote: > > No code does (or would do, or should do): > > x.counter++; > > on an "atomic_t x;" anyway. That's just an example of a general problem. No, you don't use "x.counter++". But you *do* use if (atomic_read(&x) <= 1) and loading into a register is stupid and pointless, when you could just do it as a regular memory-operand to the cmp instruction. And as far as the compiler is concerned, the problem is the 100% same: combining operations with the volatile memop. The fact is, a compiler that thinks that movl mem,reg cmpl $val,reg is any better than cmpl $val,mem is just not a very good compiler. But when talking about "volatile", that's exactly what ytou always get (and always have gotten - this is not a regression, and I doubt gcc is alone in this). Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 4:13 ` Linus Torvalds @ 2007-08-18 13:36 ` Satyam Sharma 2007-08-18 21:54 ` Paul E. McKenney 2007-08-24 12:19 ` Denys Vlasenko 2 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-18 13:36 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Lameter, Paul E. McKenney, Herbert Xu, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Fri, 17 Aug 2007, Linus Torvalds wrote: > On Sat, 18 Aug 2007, Satyam Sharma wrote: > > > > No code does (or would do, or should do): > > > > x.counter++; > > > > on an "atomic_t x;" anyway. > > That's just an example of a general problem. > > No, you don't use "x.counter++". But you *do* use > > if (atomic_read(&x) <= 1) > > and loading into a register is stupid and pointless, when you could just > do it as a regular memory-operand to the cmp instruction. True, but that makes this a bad/poor code generation issue with the compiler, not something that affects the _correctness_ of atomic ops if "volatile" is used for that counter object (as was suggested), because we'd always use the atomic_inc() etc primitives to do increments, which are always (should be!) implemented to be atomic. > And as far as the compiler is concerned, the problem is the 100% same: > combining operations with the volatile memop. > > The fact is, a compiler that thinks that > > movl mem,reg > cmpl $val,reg > > is any better than > > cmpl $val,mem > > is just not a very good compiler. Absolutely, this is definitely a bug report worth opening with gcc. And what you've said to explain this previously sounds definitely correct -- seeing "volatile" for any access does appear to just scare the hell out of gcc and makes it generate such (poor) code. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 4:13 ` Linus Torvalds 2007-08-18 13:36 ` Satyam Sharma @ 2007-08-18 21:54 ` Paul E. McKenney 2007-08-18 22:41 ` Linus Torvalds 2007-08-24 12:19 ` Denys Vlasenko 2 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-18 21:54 UTC (permalink / raw) To: Linus Torvalds Cc: Satyam Sharma, Christoph Lameter, Herbert Xu, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Fri, Aug 17, 2007 at 09:13:35PM -0700, Linus Torvalds wrote: > > > On Sat, 18 Aug 2007, Satyam Sharma wrote: > > > > No code does (or would do, or should do): > > > > x.counter++; > > > > on an "atomic_t x;" anyway. > > That's just an example of a general problem. > > No, you don't use "x.counter++". But you *do* use > > if (atomic_read(&x) <= 1) > > and loading into a register is stupid and pointless, when you could just > do it as a regular memory-operand to the cmp instruction. > > And as far as the compiler is concerned, the problem is the 100% same: > combining operations with the volatile memop. > > The fact is, a compiler that thinks that > > movl mem,reg > cmpl $val,reg > > is any better than > > cmpl $val,mem > > is just not a very good compiler. But when talking about "volatile", > that's exactly what ytou always get (and always have gotten - this is > not a regression, and I doubt gcc is alone in this). One of the gcc guys claimed that he thought that the two-instruction sequence would be faster on some x86 machines. I pointed out that there might be a concern about code size. I chose not to point out that people might also care about the other x86 machines. ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 21:54 ` Paul E. McKenney @ 2007-08-18 22:41 ` Linus Torvalds 2007-08-18 23:19 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Linus Torvalds @ 2007-08-18 22:41 UTC (permalink / raw) To: Paul E. McKenney Cc: Satyam Sharma, Christoph Lameter, Herbert Xu, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Sat, 18 Aug 2007, Paul E. McKenney wrote: > > One of the gcc guys claimed that he thought that the two-instruction > sequence would be faster on some x86 machines. I pointed out that > there might be a concern about code size. I chose not to point out > that people might also care about the other x86 machines. ;-) Some (very few) x86 uarchs do tend to prefer "load-store" like code generation, and doing a "mov [mem],reg + op reg" instead of "op [mem]" can actually be faster on some of them. Not any that are relevant today, though. Also, that has nothing to do with volatile, and should be controlled by optimization flags (like -mtune). In fact, I thought there was a separate flag to do that (ie something like "-mload-store"), but I can't find it, so maybe that's just my fevered brain.. Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 22:41 ` Linus Torvalds @ 2007-08-18 23:19 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-18 23:19 UTC (permalink / raw) To: Linus Torvalds Cc: Satyam Sharma, Christoph Lameter, Herbert Xu, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Sat, Aug 18, 2007 at 03:41:13PM -0700, Linus Torvalds wrote: > > > On Sat, 18 Aug 2007, Paul E. McKenney wrote: > > > > One of the gcc guys claimed that he thought that the two-instruction > > sequence would be faster on some x86 machines. I pointed out that > > there might be a concern about code size. I chose not to point out > > that people might also care about the other x86 machines. ;-) > > Some (very few) x86 uarchs do tend to prefer "load-store" like code > generation, and doing a "mov [mem],reg + op reg" instead of "op [mem]" can > actually be faster on some of them. Not any that are relevant today, > though. ;-) > Also, that has nothing to do with volatile, and should be controlled by > optimization flags (like -mtune). In fact, I thought there was a separate > flag to do that (ie something like "-mload-store"), but I can't find it, > so maybe that's just my fevered brain.. Good point, will suggest this if the need arises. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 4:13 ` Linus Torvalds 2007-08-18 13:36 ` Satyam Sharma 2007-08-18 21:54 ` Paul E. McKenney @ 2007-08-24 12:19 ` Denys Vlasenko 2007-08-24 17:19 ` Linus Torvalds 2 siblings, 1 reply; 333+ messages in thread From: Denys Vlasenko @ 2007-08-24 12:19 UTC (permalink / raw) To: Linus Torvalds Cc: Satyam Sharma, Christoph Lameter, Paul E. McKenney, Herbert Xu, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Saturday 18 August 2007 05:13, Linus Torvalds wrote: > On Sat, 18 Aug 2007, Satyam Sharma wrote: > > No code does (or would do, or should do): > > > > x.counter++; > > > > on an "atomic_t x;" anyway. > > That's just an example of a general problem. > > No, you don't use "x.counter++". But you *do* use > > if (atomic_read(&x) <= 1) > > and loading into a register is stupid and pointless, when you could just > do it as a regular memory-operand to the cmp instruction. It doesn't mean that (volatile int*) cast is bad, it means that current gcc is bad (or "not good enough"). IOW: instead of avoiding volatile cast, it's better to fix the compiler. > And as far as the compiler is concerned, the problem is the 100% same: > combining operations with the volatile memop. > > The fact is, a compiler that thinks that > > movl mem,reg > cmpl $val,reg > > is any better than > > cmpl $val,mem > > is just not a very good compiler. Linus, in all honesty gcc has many more cases of suboptimal code, case of "volatile" is just one of many. Off the top of my head: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=28417 unsigned v; void f(unsigned A) { v = ((unsigned long long)A) * 365384439 >> (27+32); } gcc-4.1.1 -S -Os -fomit-frame-pointer t.c f: movl $365384439, %eax mull 4(%esp) movl %edx, %eax <===== ? shrl $27, %eax movl %eax, v ret Why is it moving %edx to %eax? gcc-4.2.1 -S -Os -fomit-frame-pointer t.c f: movl $365384439, %eax mull 4(%esp) movl %edx, %eax <===== ? xorl %edx, %edx <===== ??! shrl $27, %eax movl %eax, v ret Progress... Now we also zero out %edx afterwards for no apparent reason. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-24 12:19 ` Denys Vlasenko @ 2007-08-24 17:19 ` Linus Torvalds 0 siblings, 0 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-24 17:19 UTC (permalink / raw) To: Denys Vlasenko Cc: Satyam Sharma, Christoph Lameter, Paul E. McKenney, Herbert Xu, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, schwidefsky, Chris Snook, davem, wensong, wjiang On Fri, 24 Aug 2007, Denys Vlasenko wrote: > > > No, you don't use "x.counter++". But you *do* use > > > > if (atomic_read(&x) <= 1) > > > > and loading into a register is stupid and pointless, when you could just > > do it as a regular memory-operand to the cmp instruction. > > It doesn't mean that (volatile int*) cast is bad, it means that current gcc > is bad (or "not good enough"). IOW: instead of avoiding volatile cast, > it's better to fix the compiler. I would agree that fixing the compiler in this case would be a good thing, even quite regardless of any "atomic_read()" discussion. I just have a strong suspicion that "volatile" performance is so low down the list of any C compiler persons interest, that it's never going to happen. And quite frankly, I cannot blame the gcc guys for it. That's especially as "volatile" really isn't a very good feature of the C language, and is likely to get *less* interesting rather than more (as user space starts to be more and more threaded, "volatile" gets less and less useful. [ Ie, currently, I think you can validly use "volatile" in a "sigatomic_t" kind of way, where there is a single thread, but with asynchronous events. In that kind of situation, I think it's probably useful. But once you get multiple threads, it gets pointless. Sure: you could use "volatile" together with something like Dekker's or Peterson's algorithm that doesn't depend on cache coherency (that's basically what the C "volatile" keyword approximates: not atomic accesses, but *uncached* accesses! But let's face it, that's way past insane. ] So I wouldn't expect "volatile" to ever really generate better code. It might happen as a side effect of other improvements (eg, I might hope that the SSA work would eventually lead to gcc having a much better defined model of valid optimizations, and maybe better code generation for volatile accesses fall out cleanly out of that), but in the end, it's such an ugly special case in C, and so seldom used, that I wouldn't depend on it. > Linus, in all honesty gcc has many more cases of suboptimal code, > case of "volatile" is just one of many. Well, the thing is, quite often, many of those "suboptimal code" generations fall into two distinct classes: - complex C code. I can't really blame the compiler too much for this. Some things are *hard* to optimize, and for various scalability reasons, you often end up having limits in the compiler where it doesn't even _try_ doing certain optimizations if you have excessive complexity. - bad register allocation. Register allocation really is hard, and sometimes gcc just does the "obviously wrong" thing, and you end up having totally unnecessary spills. > Off the top of my head: Yes, "unsigned long long" with x86 has always generated atrocious code. In fact, I would say that historically it was really *really* bad. These days, gcc actually does a pretty good job, but I'm not surprised that it's still quite possible to find cases where it did some optimization (in this case, apparently noticing that "shift by >= 32 bits" causes the low register to be pointless) and then missed *another* optimization (better register use) because that optimization had been done *before* the first optimization was done. That's a *classic* example of compiler code generation issues, and quite frankly, I think that's very different from the issue of "volatile". Quite frankly, I'd like there to be more competition in the open source compiler game, and that might cause some upheavals, but on the whole, gcc actually does a pretty damn good job. Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 1:24 ` Christoph Lameter 2007-08-18 1:41 ` Satyam Sharma @ 2007-08-18 21:56 ` Paul E. McKenney 2007-08-20 13:31 ` Chris Snook 2 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-18 21:56 UTC (permalink / raw) To: Christoph Lameter Cc: Herbert Xu, Linus Torvalds, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, schwidefsky, Chris Snook, davem, wensong, wjiang On Fri, Aug 17, 2007 at 06:24:15PM -0700, Christoph Lameter wrote: > On Fri, 17 Aug 2007, Paul E. McKenney wrote: > > > On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote: > > > On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote: > > > > > > > > gcc bugzilla bug #33102, for whatever that ends up being worth. ;-) > > > > > > I had totally forgotten that I'd already filed that bug more > > > than six years ago until they just closed yours as a duplicate > > > of mine :) > > > > > > Good luck in getting it fixed! > > > > Well, just got done re-opening it for the third time. And a local > > gcc community member advised me not to give up too easily. But I > > must admit that I am impressed with the speed that it was identified > > as duplicate. > > > > Should be entertaining! ;-) > > Right. ROTFL... volatile actually breaks atomic_t instead of making it > safe. x++ becomes a register load, increment and a register store. Without > volatile we can increment the memory directly. It seems that volatile > requires that the variable is loaded into a register first and then > operated upon. Understandable when you think about volatile being used to > access memory mapped I/O registers where a RMW operation could be > problematic. > > See http://gcc.gnu.org/bugzilla/show_bug.cgi?id=3506 Yep. The initial reaction was in fact to close my bug as a duplicate of 3506. But I was not asking for atomicity, but rather for smaller code to be generated, so I reopened it. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 1:24 ` Christoph Lameter 2007-08-18 1:41 ` Satyam Sharma 2007-08-18 21:56 ` Paul E. McKenney @ 2007-08-20 13:31 ` Chris Snook 2007-08-20 22:04 ` Segher Boessenkool 2 siblings, 1 reply; 333+ messages in thread From: Chris Snook @ 2007-08-20 13:31 UTC (permalink / raw) To: Christoph Lameter Cc: Paul E. McKenney, Herbert Xu, Linus Torvalds, Nick Piggin, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, schwidefsky, davem, wensong, wjiang Christoph Lameter wrote: > On Fri, 17 Aug 2007, Paul E. McKenney wrote: > >> On Sat, Aug 18, 2007 at 08:09:13AM +0800, Herbert Xu wrote: >>> On Fri, Aug 17, 2007 at 04:59:12PM -0700, Paul E. McKenney wrote: >>>> gcc bugzilla bug #33102, for whatever that ends up being worth. ;-) >>> I had totally forgotten that I'd already filed that bug more >>> than six years ago until they just closed yours as a duplicate >>> of mine :) >>> >>> Good luck in getting it fixed! >> Well, just got done re-opening it for the third time. And a local >> gcc community member advised me not to give up too easily. But I >> must admit that I am impressed with the speed that it was identified >> as duplicate. >> >> Should be entertaining! ;-) > > Right. ROTFL... volatile actually breaks atomic_t instead of making it > safe. x++ becomes a register load, increment and a register store. Without > volatile we can increment the memory directly. It seems that volatile > requires that the variable is loaded into a register first and then > operated upon. Understandable when you think about volatile being used to > access memory mapped I/O registers where a RMW operation could be > problematic. So, if we want consistent behavior, we're pretty much screwed unless we use inline assembler everywhere? -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 13:31 ` Chris Snook @ 2007-08-20 22:04 ` Segher Boessenkool 2007-08-20 22:48 ` Russell King 0 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-20 22:04 UTC (permalink / raw) To: Chris Snook Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> Right. ROTFL... volatile actually breaks atomic_t instead of making >> it safe. x++ becomes a register load, increment and a register store. >> Without volatile we can increment the memory directly. It seems that >> volatile requires that the variable is loaded into a register first >> and then operated upon. Understandable when you think about volatile >> being used to access memory mapped I/O registers where a RMW >> operation could be problematic. > > So, if we want consistent behavior, we're pretty much screwed unless > we use inline assembler everywhere? Nah, this whole argument is flawed -- "without volatile" we still *cannot* "increment the memory directly". On x86, you need a lock prefix; on other archs, some other mechanism to make the memory increment an *atomic* memory increment. And no, RMW on MMIO isn't "problematic" at all, either. An RMW op is a read op, a modify op, and a write op, all rolled into one opcode. But three actual operations. The advantages of asm code for atomic_{read,set} are: 1) all the other atomic ops are implemented that way already; 2) you have full control over the asm insns selected, in particular, you can guarantee you *do* get an atomic op; 3) you don't need to use "volatile <data>" which generates not-all-that-good code on archs like x86, and we want to get rid of it anyway since it is problematic in many ways; 4) you don't need to use *(volatile <type>*)&<data>, which a) doesn't exist in C; b) isn't documented or supported in GCC; c) has a recent history of bugginess; d) _still uses volatile objects_; e) _still_ is problematic in almost all those same ways as in 3); 5) you can mix atomic and non-atomic accesses to the atomic_t, which you cannot with the other alternatives. The only disadvantage I know of is potentially slightly worse instruction scheduling. This is a generic asm() problem: GCC cannot see what actual insns are inside the asm() block. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 22:04 ` Segher Boessenkool @ 2007-08-20 22:48 ` Russell King 2007-08-20 23:02 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Russell King @ 2007-08-20 22:48 UTC (permalink / raw) To: Segher Boessenkool Cc: Chris Snook, Christoph Lameter, Paul Mackerras, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Tue, Aug 21, 2007 at 12:04:17AM +0200, Segher Boessenkool wrote: > And no, RMW on MMIO isn't "problematic" at all, either. > > An RMW op is a read op, a modify op, and a write op, all rolled > into one opcode. But three actual operations. Maybe for some CPUs, but not all. ARM for instance can't use the load exclusive and store exclusive instructions to MMIO space. This means placing atomic_t or bitops into MMIO space is a definite no-go on ARM. It breaks. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 22:48 ` Russell King @ 2007-08-20 23:02 ` Segher Boessenkool 2007-08-21 0:05 ` Paul E. McKenney 2007-08-21 7:05 ` Russell King 0 siblings, 2 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-20 23:02 UTC (permalink / raw) To: Russell King Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> And no, RMW on MMIO isn't "problematic" at all, either. >> >> An RMW op is a read op, a modify op, and a write op, all rolled >> into one opcode. But three actual operations. > > Maybe for some CPUs, but not all. ARM for instance can't use the > load exclusive and store exclusive instructions to MMIO space. Sure, your CPU doesn't have RMW instructions -- how to emulate those if you don't have them is a totally different thing. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 23:02 ` Segher Boessenkool @ 2007-08-21 0:05 ` Paul E. McKenney 2007-08-21 7:08 ` Russell King 2007-08-21 7:05 ` Russell King 1 sibling, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-21 0:05 UTC (permalink / raw) To: Segher Boessenkool Cc: Russell King, Christoph Lameter, Paul Mackerras, heiko.carstens, horms, linux-kernel, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Tue, Aug 21, 2007 at 01:02:01AM +0200, Segher Boessenkool wrote: > >>And no, RMW on MMIO isn't "problematic" at all, either. > >> > >>An RMW op is a read op, a modify op, and a write op, all rolled > >>into one opcode. But three actual operations. > > > >Maybe for some CPUs, but not all. ARM for instance can't use the > >load exclusive and store exclusive instructions to MMIO space. > > Sure, your CPU doesn't have RMW instructions -- how to emulate > those if you don't have them is a totally different thing. I thought that ARM's load exclusive and store exclusive instructions were its equivalent of LL and SC, which RISC machines typically use to build atomic sequences of instructions -- and which normally cannot be applied to MMIO space. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 0:05 ` Paul E. McKenney @ 2007-08-21 7:08 ` Russell King 0 siblings, 0 replies; 333+ messages in thread From: Russell King @ 2007-08-21 7:08 UTC (permalink / raw) To: Paul E. McKenney Cc: Segher Boessenkool, Christoph Lameter, Paul Mackerras, heiko.carstens, horms, linux-kernel, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Mon, Aug 20, 2007 at 05:05:18PM -0700, Paul E. McKenney wrote: > On Tue, Aug 21, 2007 at 01:02:01AM +0200, Segher Boessenkool wrote: > > >>And no, RMW on MMIO isn't "problematic" at all, either. > > >> > > >>An RMW op is a read op, a modify op, and a write op, all rolled > > >>into one opcode. But three actual operations. > > > > > >Maybe for some CPUs, but not all. ARM for instance can't use the > > >load exclusive and store exclusive instructions to MMIO space. > > > > Sure, your CPU doesn't have RMW instructions -- how to emulate > > those if you don't have them is a totally different thing. > > I thought that ARM's load exclusive and store exclusive instructions > were its equivalent of LL and SC, which RISC machines typically use to > build atomic sequences of instructions -- and which normally cannot be > applied to MMIO space. Absolutely correct. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 23:02 ` Segher Boessenkool 2007-08-21 0:05 ` Paul E. McKenney @ 2007-08-21 7:05 ` Russell King 2007-08-21 9:33 ` Paul Mackerras 2007-08-21 14:39 ` Segher Boessenkool 1 sibling, 2 replies; 333+ messages in thread From: Russell King @ 2007-08-21 7:05 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Tue, Aug 21, 2007 at 01:02:01AM +0200, Segher Boessenkool wrote: > >>And no, RMW on MMIO isn't "problematic" at all, either. > >> > >>An RMW op is a read op, a modify op, and a write op, all rolled > >>into one opcode. But three actual operations. > > > >Maybe for some CPUs, but not all. ARM for instance can't use the > >load exclusive and store exclusive instructions to MMIO space. > > Sure, your CPU doesn't have RMW instructions -- how to emulate > those if you don't have them is a totally different thing. Let me say it more clearly: On ARM, it is impossible to perform atomic operations on MMIO space. -- Russell King Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/ maintainer of: ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 7:05 ` Russell King @ 2007-08-21 9:33 ` Paul Mackerras 2007-08-21 11:37 ` Andi Kleen 2007-08-21 14:48 ` Segher Boessenkool 2007-08-21 14:39 ` Segher Boessenkool 1 sibling, 2 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-21 9:33 UTC (permalink / raw) To: Russell King Cc: Segher Boessenkool, Christoph Lameter, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang Russell King writes: > Let me say it more clearly: On ARM, it is impossible to perform atomic > operations on MMIO space. Actually, no one is suggesting that we try to do that at all. The discussion about RMW ops on MMIO space started with a comment attributed to the gcc developers that one reason why gcc on x86 doesn't use instructions that do RMW ops on volatile variables is that volatile is used to mark MMIO addresses, and there was some uncertainty about whether (non-atomic) RMW ops on x86 could be used on MMIO. This is in regard to the question about why gcc on x86 always moves a volatile variable into a register before doing anything to it. So the whole discussion is irrelevant to ARM, PowerPC and any other architecture except x86[-64]. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 9:33 ` Paul Mackerras @ 2007-08-21 11:37 ` Andi Kleen 2007-08-21 14:48 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Andi Kleen @ 2007-08-21 11:37 UTC (permalink / raw) To: Paul Mackerras Cc: Russell King, Segher Boessenkool, Christoph Lameter, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Tue, Aug 21, 2007 at 07:33:49PM +1000, Paul Mackerras wrote: > So the whole discussion is irrelevant to ARM, PowerPC and any other > architecture except x86[-64]. It's even irrelevant on x86 because all modifying operations on atomic_t are coded in inline assembler and will always be RMW no matter if atomic_t is volatile or not. [ignoring atomic_set(x, atomic_read(x) + 1) which nobody should do] The only issue is if atomic_t should have a implicit barrier or not. My personal opinion is yes -- better safe than sorry. And any code impact it may have is typically dwarved by the next cache miss anyways, so it doesn't matter much. -Andi ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 9:33 ` Paul Mackerras 2007-08-21 11:37 ` Andi Kleen @ 2007-08-21 14:48 ` Segher Boessenkool 2007-08-21 16:16 ` Paul E. McKenney 1 sibling, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-21 14:48 UTC (permalink / raw) To: Paul Mackerras Cc: Russell King, Christoph Lameter, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> Let me say it more clearly: On ARM, it is impossible to perform atomic >> operations on MMIO space. > > Actually, no one is suggesting that we try to do that at all. > > The discussion about RMW ops on MMIO space started with a comment > attributed to the gcc developers that one reason why gcc on x86 > doesn't use instructions that do RMW ops on volatile variables is that > volatile is used to mark MMIO addresses, and there was some > uncertainty about whether (non-atomic) RMW ops on x86 could be used on > MMIO. This is in regard to the question about why gcc on x86 always > moves a volatile variable into a register before doing anything to it. This question is GCC PR33102, which was incorrectly closed as a duplicate of PR3506 -- and *that* PR was closed because its reporter seemed to claim the GCC generated code for an increment on a volatile (namely, three machine instructions: load, modify, store) was incorrect, and it has to be one machine instruction. > So the whole discussion is irrelevant to ARM, PowerPC and any other > architecture except x86[-64]. And even there, it's not something the kernel can take advantage of before GCC 4.4 is in widespread use, if then. Let's move on. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 14:48 ` Segher Boessenkool @ 2007-08-21 16:16 ` Paul E. McKenney 2007-08-21 22:51 ` Valdis.Kletnieks 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-21 16:16 UTC (permalink / raw) To: Segher Boessenkool Cc: Paul Mackerras, Russell King, Christoph Lameter, heiko.carstens, horms, linux-kernel, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Tue, Aug 21, 2007 at 04:48:51PM +0200, Segher Boessenkool wrote: > >>Let me say it more clearly: On ARM, it is impossible to perform atomic > >>operations on MMIO space. > > > >Actually, no one is suggesting that we try to do that at all. > > > >The discussion about RMW ops on MMIO space started with a comment > >attributed to the gcc developers that one reason why gcc on x86 > >doesn't use instructions that do RMW ops on volatile variables is that > >volatile is used to mark MMIO addresses, and there was some > >uncertainty about whether (non-atomic) RMW ops on x86 could be used on > >MMIO. This is in regard to the question about why gcc on x86 always > >moves a volatile variable into a register before doing anything to it. > > This question is GCC PR33102, which was incorrectly closed as a > duplicate > of PR3506 -- and *that* PR was closed because its reporter seemed to > claim the GCC generated code for an increment on a volatile (namely, > three > machine instructions: load, modify, store) was incorrect, and it has to > be one machine instruction. > > >So the whole discussion is irrelevant to ARM, PowerPC and any other > >architecture except x86[-64]. > > And even there, it's not something the kernel can take advantage of > before GCC 4.4 is in widespread use, if then. Let's move on. I agree that instant gratification is hard to come by when synching up compiler and kernel versions. Nonetheless, it should be possible to create APIs that are are conditioned on the compiler version. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 16:16 ` Paul E. McKenney @ 2007-08-21 22:51 ` Valdis.Kletnieks 2007-08-22 0:50 ` Paul E. McKenney 2007-08-22 21:38 ` Adrian Bunk 0 siblings, 2 replies; 333+ messages in thread From: Valdis.Kletnieks @ 2007-08-21 22:51 UTC (permalink / raw) To: paulmck Cc: Segher Boessenkool, Paul Mackerras, Russell King, Christoph Lameter, heiko.carstens, horms, linux-kernel, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang [-- Attachment #1: Type: text/plain, Size: 557 bytes --] On Tue, 21 Aug 2007 09:16:43 PDT, "Paul E. McKenney" said: > I agree that instant gratification is hard to come by when synching > up compiler and kernel versions. Nonetheless, it should be possible > to create APIs that are are conditioned on the compiler version. We've tried that, sort of. See the mess surrounding the whole extern/static/inline/__whatever boondogle, which seems to have changed semantics in every single gcc release since 2.95 or so. And recently mention was made that gcc4.4 will have *new* semantics in this area. Yee. Hah. [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 22:51 ` Valdis.Kletnieks @ 2007-08-22 0:50 ` Paul E. McKenney 2007-08-22 21:38 ` Adrian Bunk 1 sibling, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-22 0:50 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Segher Boessenkool, Paul Mackerras, Russell King, Christoph Lameter, heiko.carstens, horms, linux-kernel, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Tue, Aug 21, 2007 at 06:51:16PM -0400, Valdis.Kletnieks@vt.edu wrote: > On Tue, 21 Aug 2007 09:16:43 PDT, "Paul E. McKenney" said: > > > I agree that instant gratification is hard to come by when synching > > up compiler and kernel versions. Nonetheless, it should be possible > > to create APIs that are are conditioned on the compiler version. > > We've tried that, sort of. See the mess surrounding the whole > extern/static/inline/__whatever boondogle, which seems to have > changed semantics in every single gcc release since 2.95 or so. > > And recently mention was made that gcc4.4 will have *new* semantics > in this area. Yee. Hah. ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 22:51 ` Valdis.Kletnieks 2007-08-22 0:50 ` Paul E. McKenney @ 2007-08-22 21:38 ` Adrian Bunk 1 sibling, 0 replies; 333+ messages in thread From: Adrian Bunk @ 2007-08-22 21:38 UTC (permalink / raw) To: Valdis.Kletnieks Cc: paulmck, Segher Boessenkool, Paul Mackerras, Russell King, Christoph Lameter, heiko.carstens, horms, linux-kernel, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Tue, Aug 21, 2007 at 06:51:16PM -0400, Valdis.Kletnieks@vt.edu wrote: > On Tue, 21 Aug 2007 09:16:43 PDT, "Paul E. McKenney" said: > > > I agree that instant gratification is hard to come by when synching > > up compiler and kernel versions. Nonetheless, it should be possible > > to create APIs that are are conditioned on the compiler version. > > We've tried that, sort of. See the mess surrounding the whole > extern/static/inline/__whatever boondogle, which seems to have > changed semantics in every single gcc release since 2.95 or so. >... There is exactly one semantics change in gcc in this area, and that is the change of the "extern inline" semantics in gcc 4.3 to the C99 semantics. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 7:05 ` Russell King 2007-08-21 9:33 ` Paul Mackerras @ 2007-08-21 14:39 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-21 14:39 UTC (permalink / raw) To: Russell King Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, linux-kernel, Paul E. McKenney, ak, netdev, cfriesen, akpm, rpjday, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >>>> And no, RMW on MMIO isn't "problematic" at all, either. >>>> >>>> An RMW op is a read op, a modify op, and a write op, all rolled >>>> into one opcode. But three actual operations. >>> >>> Maybe for some CPUs, but not all. ARM for instance can't use the >>> load exclusive and store exclusive instructions to MMIO space. >> >> Sure, your CPU doesn't have RMW instructions -- how to emulate >> those if you don't have them is a totally different thing. > > Let me say it more clearly: On ARM, it is impossible to perform atomic > operations on MMIO space. It's all completely beside the point, see the other subthread, but... Yeah, you can't do LL/SC to MMIO space; ARM isn't alone in that. You could still implement atomic operations on MMIO space by taking a lock elsewhere, in normal cacheable memory space. Why you would do this is a separate question, you probably don't want it :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:16 ` Paul Mackerras 2007-08-17 3:32 ` Nick Piggin @ 2007-08-17 3:42 ` Linus Torvalds 2007-08-17 5:18 ` Paul E. McKenney ` (4 more replies) 1 sibling, 5 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-17 3:42 UTC (permalink / raw) To: Paul Mackerras Cc: Nick Piggin, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Fri, 17 Aug 2007, Paul Mackerras wrote: > > I'm really surprised it's as much as a few K. I tried it on powerpc > and it only saved 40 bytes (10 instructions) for a G5 config. One of the things that "volatile" generally screws up is a simple volatile int i; i++; which a compiler will generally get horribly, horribly wrong. In a reasonable world, gcc should just make that be (on x86) addl $1,i(%rip) on x86-64, which is indeed what it does without the volatile. But with the volatile, the compiler gets really nervous, and doesn't dare do it in one instruction, and thus generates crap like movl i(%rip), %eax addl $1, %eax movl %eax, i(%rip) instead. For no good reason, except that "volatile" just doesn't have any good/clear semantics for the compiler, so most compilers will just make it be "I will not touch this access in any way, shape, or form". Including even trivially correct instruction optimization/combination. This is one of the reasons why we should never use "volatile". It pessimises code generation for no good reason - just because compilers don't know what the heck it even means! Now, people don't do "i++" on atomics (you'd use "atomic_inc()" for that), but people *do* do things like if (atomic_read(..) <= 1) .. On ppc, things like that probably don't much matter. But on x86, it makes a *huge* difference whether you do movl i(%rip),%eax cmpl $1,%eax or if you can just use the value directly for the operation, like this: cmpl $1,i(%rip) which is again a totally obvious and totally safe optimization, but is (again) something that gcc doesn't dare do, since "i" is volatile. In other words: "volatile" is a horribly horribly bad way of doing things, because it generates *worse*code*, for no good reason. You just don't see it on powerpc, because it's already a load-store architecture, so there is no "good code" for doing direct-to-memory operations. Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:42 ` Linus Torvalds @ 2007-08-17 5:18 ` Paul E. McKenney 2007-08-17 5:56 ` Satyam Sharma ` (3 subsequent siblings) 4 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 5:18 UTC (permalink / raw) To: Linus Torvalds Cc: Paul Mackerras, Nick Piggin, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Thu, Aug 16, 2007 at 08:42:23PM -0700, Linus Torvalds wrote: > > > On Fri, 17 Aug 2007, Paul Mackerras wrote: > > > > I'm really surprised it's as much as a few K. I tried it on powerpc > > and it only saved 40 bytes (10 instructions) for a G5 config. > > One of the things that "volatile" generally screws up is a simple > > volatile int i; > > i++; > > which a compiler will generally get horribly, horribly wrong. > > In a reasonable world, gcc should just make that be (on x86) > > addl $1,i(%rip) > > on x86-64, which is indeed what it does without the volatile. But with the > volatile, the compiler gets really nervous, and doesn't dare do it in one > instruction, and thus generates crap like > > movl i(%rip), %eax > addl $1, %eax > movl %eax, i(%rip) Blech. Sounds like a chat with some gcc people is in order. Will see what I can do. Thanx, Paul > instead. For no good reason, except that "volatile" just doesn't have any > good/clear semantics for the compiler, so most compilers will just make it > be "I will not touch this access in any way, shape, or form". Including > even trivially correct instruction optimization/combination. > > This is one of the reasons why we should never use "volatile". It > pessimises code generation for no good reason - just because compilers > don't know what the heck it even means! > > Now, people don't do "i++" on atomics (you'd use "atomic_inc()" for that), > but people *do* do things like > > if (atomic_read(..) <= 1) > .. > > On ppc, things like that probably don't much matter. But on x86, it makes > a *huge* difference whether you do > > movl i(%rip),%eax > cmpl $1,%eax > > or if you can just use the value directly for the operation, like this: > > cmpl $1,i(%rip) > > which is again a totally obvious and totally safe optimization, but is > (again) something that gcc doesn't dare do, since "i" is volatile. > > In other words: "volatile" is a horribly horribly bad way of doing things, > because it generates *worse*code*, for no good reason. You just don't see > it on powerpc, because it's already a load-store architecture, so there is > no "good code" for doing direct-to-memory operations. > > Linus > - > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:42 ` Linus Torvalds 2007-08-17 5:18 ` Paul E. McKenney @ 2007-08-17 5:56 ` Satyam Sharma 2007-08-17 7:26 ` Nick Piggin 2007-08-17 22:49 ` Segher Boessenkool 2007-08-17 6:42 ` Geert Uytterhoeven ` (2 subsequent siblings) 4 siblings, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 5:56 UTC (permalink / raw) To: Linus Torvalds Cc: Paul Mackerras, Nick Piggin, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Hi Linus, [ and others; I think there's a communication gap in a lot of this thread, and a little summary would be useful. Hence this posting. ] On Thu, 16 Aug 2007, Linus Torvalds wrote: > On Fri, 17 Aug 2007, Paul Mackerras wrote: > > > > I'm really surprised it's as much as a few K. I tried it on powerpc > > and it only saved 40 bytes (10 instructions) for a G5 config. > > One of the things that "volatile" generally screws up is a simple > > volatile int i; > > i++; > > which a compiler will generally get horribly, horribly wrong. > > [...] For no good reason, except that "volatile" just doesn't have any > good/clear semantics for the compiler, so most compilers will just make it > be "I will not touch this access in any way, shape, or form". Including > even trivially correct instruction optimization/combination. > > This is one of the reasons why we should never use "volatile". It > pessimises code generation for no good reason - just because compilers > don't know what the heck it even means! > [...] > In other words: "volatile" is a horribly horribly bad way of doing things, > because it generates *worse*code*, for no good reason. You just don't see > it on powerpc, because it's already a load-store architecture, so there is > no "good code" for doing direct-to-memory operations. True, and I bet *everybody* on this list has already agreed for a very long time that using "volatile" to type-qualify the _declaration_ of an object itself as being horribly bad (taste-wise, code-generation-wise, often even buggy for sitations where real CPU barriers should've been used instead). However, the discussion on this thread (IIRC) began with only "giving volatility semantics" to atomic ops. Now that is different, and may not require the use the "volatile" keyword (at least not in the declaration of the object) itself. Sadly, most arch's *still* do type-qualify the declaration of the "counter" member of atomic_t as "volatile". This is probably a historic hangover, and I suspect not yet rectified because of lethargy. Anyway, some of the variants I can think of are: [1] #define atomic_read_volatile(v) \ ({ \ forget((v)->counter); \ ((v)->counter); \ }) where: #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) [ This is exactly equivalent to using "+m" in the constraints, as recently explained on a GCC list somewhere, in response to the patch in my bitops series a few weeks back where I thought "+m" was bogus. ] [2] #define atomic_read_volatile(v) (*(volatile int *)&(v)->counter) This is something that does work. It has reasonably good semantics guaranteed by the C standard in conjunction with how GCC currently behaves (and how it has behaved for all supported versions). I haven't checked if generates much different code than the first variant above, (it probably will generate similar code to just declaring the object as volatile, but would still be better in terms of code-clarity and taste, IMHO), but in any case, we should pick whichever of these variants works for us and generates good code. [3] static inline int atomic_read_volatile(atomic_t *v) { ... arch-dependent __asm__ __volatile__ stuff ... } I can reasonably bet this variant would often generate worse code than at least the variant "[1]" above. Now, why do we even require these "volatility" semantics variants? Note, "volatility" semantics *know* / assume that it can have a meaning _only_ as far as the compiler, so atomic_read_volatile() doesn't really care reading stale values from the cache for certain non-x86 archs, etc. The first argument is "safety": Use of atomic_read() (possibly in conjunction with other atomic ops) in a lot of code out there in the kernel *assumes* the compiler will not optimize away those ops. (which is possible given current definitions of atomic_set and atomic_read on archs such as x86 in present code). An additional argument that builds on this one says that by ensuring the compiler will not elid or coalesce these ops, we could even avoid potential heisenbugs in the future. However, there is a counter-argument: As Herbert Xu has often been making the point, there is *no* bug out there involving "atomic_read" in busy-while-loops that should not have a compiler barrier (or cpu_relax() in fact) anyway. As for non-busy-loops, they would invariable call schedule() at some point (possibly directly) and thus have an "implicit" compiler barrier by virtue of calling out a function that is not in scope of the current compilation unit (although users in sched.c itself would probably require an explicit compiler barrier). The second pro-volatility-in-atomic-ops argument is performance: (surprise!) Using a full memory clobber compiler barrier in busy loops will disqualify optimizations for loop invariants so it probably makes sense to *only* make the compiler forget *that* particular address of the atomic counter object, and none other. All 3 variants above would work nicely here. So the final idea may be to have a cpu_relax_no_barrier() variant as a rep;nop (pause) *without* an included full memory clobber, and replace a lot of kernel busy-while-loops out there with: - cpu_relax(); + cpu_relax_no_barrier(); + forget(x); or may be just: - cpu_relax(); + cpu_relax_no_barrier(); because the "forget" / "volatility" / specific-variable-compiler-barrier could be made implicit inside the atomic ops themselves. This could especially make a difference for register-rich CPUs (probably not x86) where using a full memory clobber will disqualify a hell lot of compiler optimizations for loop-invariants. On x86 itself, cpu_relax_no_barrier() could be: #define cpu_relax_no_barrier() __asm__ __volatile__ ("rep;nop":::); and still continue to do its job as it is doing presently. However, there is still a counter-argument: As Herbert Xu and Christoph Lameter have often been saying, giving "volatility" semantics to the atomic ops will disqualify compiler optimizations such as eliding / coalescing of atomic ops, etc, and probably some sections of code in the kernel (Christoph mentioned code in SLUB, and I saw such code in sched) benefit from such optimizations. Paul Mackerras has, otoh, mentioned that a lot of such places probably don't need (or shouldn't use) atomic ops in the first place. Alternatively, such callsites should probably just cache the atomic_read in a local variable (which compiler could just as well make a register) explicitly, and repeating atomic_read() isn't really necessary. There could still be legitimate uses of atomic ops that don't care about them being elided / coalesced, but given the loop-invariant-optimization benefit, personally, I do see some benefit in the use of defining atomic ops variants with "volatility" semantics (for only the atomic counter object) but also having a non-volatile atomic ops API side-by-side for performance critical users (probably sched, slub) that may require that. Possibly, one of the two APIs above could turn out to be redundant, but that's still very much the issue of debate presently. Satyam [ Sorry if I missed anything important, but this thread has been long and noisy, although I've tried to keep up ... ] ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 5:56 ` Satyam Sharma @ 2007-08-17 7:26 ` Nick Piggin 2007-08-17 8:47 ` Satyam Sharma 2007-08-17 22:49 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-17 7:26 UTC (permalink / raw) To: Satyam Sharma Cc: Linus Torvalds, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Satyam Sharma wrote: > #define atomic_read_volatile(v) \ > ({ \ > forget((v)->counter); \ > ((v)->counter); \ > }) > > where: *vomit* :) Not only do I hate the keyword volatile, but the barrier is only a one-sided affair so its probable this is going to have slightly different allowed reorderings than a real volatile access. Also, why would you want to make these insane accessors for atomic_t types? Just make sure everybody knows the basics of barriers, and they can apply that knowledge to atomic_t and all other lockless memory accesses as well. > #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) I like order(x) better, but it's not the most perfect name either. -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 7:26 ` Nick Piggin @ 2007-08-17 8:47 ` Satyam Sharma 2007-08-17 9:15 ` Nick Piggin 2007-08-17 9:48 ` Paul Mackerras 0 siblings, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 8:47 UTC (permalink / raw) To: Nick Piggin Cc: Linus Torvalds, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > > > #define atomic_read_volatile(v) \ > > ({ \ > > forget((v)->counter); \ > > ((v)->counter); \ > > }) > > > > where: > > *vomit* :) I wonder if this'll generate smaller and better code than _both_ the other atomic_read_volatile() variants. Would need to build allyesconfig on lots of diff arch's etc to test the theory though. > Not only do I hate the keyword volatile, but the barrier is only a > one-sided affair so its probable this is going to have slightly > different allowed reorderings than a real volatile access. True ... > Also, why would you want to make these insane accessors for atomic_t > types? Just make sure everybody knows the basics of barriers, and they > can apply that knowledge to atomic_t and all other lockless memory > accesses as well. Code that looks like: while (!atomic_read(&v)) { ... cpu_relax_no_barrier(); forget(v.counter); ^^^^^^^^ } would be uglier. Also think about code such as: a = atomic_read(); if (!a) do_something(); forget(); a = atomic_read(); ... /* some code that depends on value of a, obviously */ forget(); a = atomic_read(); ... So much explicit sprinkling of "forget()" looks ugly. atomic_read_volatile() on the other hand, looks neater. The "_volatile()" suffix makes it also no less explicit than an explicit barrier-like macro that this primitive is something "special", for code clarity purposes. > > #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) > > I like order(x) better, but it's not the most perfect name either. forget(x) is just a stupid-placeholder-for-a-better-name. order(x) sounds good but we could leave quibbling about function or macro names for later, this thread is noisy as it is :-) ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:47 ` Satyam Sharma @ 2007-08-17 9:15 ` Nick Piggin 2007-08-17 10:12 ` Satyam Sharma 2007-08-17 9:48 ` Paul Mackerras 1 sibling, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-17 9:15 UTC (permalink / raw) To: Satyam Sharma Cc: Linus Torvalds, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Satyam Sharma wrote: > > On Fri, 17 Aug 2007, Nick Piggin wrote: >>Also, why would you want to make these insane accessors for atomic_t >>types? Just make sure everybody knows the basics of barriers, and they >>can apply that knowledge to atomic_t and all other lockless memory >>accesses as well. > > > Code that looks like: > > while (!atomic_read(&v)) { > ... > cpu_relax_no_barrier(); > forget(v.counter); > ^^^^^^^^ > } > > would be uglier. Also think about code such as: I think they would both be equally ugly, but the atomic_read_volatile variant would be more prone to subtle bugs because of the weird implementation. And it would be more ugly than introducing an order(x) statement for all memory operations, and adding an order_atomic() wrapper for it for atomic types. > a = atomic_read(); > if (!a) > do_something(); > > forget(); > a = atomic_read(); > ... /* some code that depends on value of a, obviously */ > > forget(); > a = atomic_read(); > ... > > So much explicit sprinkling of "forget()" looks ugly. Firstly, why is it ugly? It's nice because of those nice explicit statements there that give us a good heads up and would have some comments attached to them (also, lack of the word "volatile" is always a plus). Secondly, what sort of code would do such a thing? In most cases, it is probably riddled with bugs anyway (unless it is doing a really specific sequence of interrupts or something, but in that case it is very likely to either require locking or busy waits anyway -> ie. barriers). > on the other hand, looks neater. The "_volatile()" suffix makes it also > no less explicit than an explicit barrier-like macro that this primitive > is something "special", for code clarity purposes. Just don't use the word volatile, and have barriers both before and after the memory operation, and I'm OK with it. I don't see the point though, when you could just have a single barrier(x) barrier function defined for all memory locations, rather than this odd thing that only works for atomics (and would have to be duplicated for atomic_set. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 9:15 ` Nick Piggin @ 2007-08-17 10:12 ` Satyam Sharma 2007-08-17 12:14 ` Nick Piggin 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 10:12 UTC (permalink / raw) To: Nick Piggin Cc: Linus Torvalds, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > > > > On Fri, 17 Aug 2007, Nick Piggin wrote: > > > > Also, why would you want to make these insane accessors for atomic_t > > > types? Just make sure everybody knows the basics of barriers, and they > > > can apply that knowledge to atomic_t and all other lockless memory > > > accesses as well. > > > > > > Code that looks like: > > > > while (!atomic_read(&v)) { > > ... > > cpu_relax_no_barrier(); > > forget(v.counter); > > ^^^^^^^^ > > } > > > > would be uglier. Also think about code such as: > > I think they would both be equally ugly, You think both these are equivalent in terms of "looks": | while (!atomic_read(&v)) { | while (!atomic_read_xxx(&v)) { ... | ... cpu_relax_no_barrier(); | cpu_relax_no_barrier(); order_atomic(&v); | } } | (where order_atomic() is an atomic_t specific wrapper as you mentioned below) ? Well, taste varies, but ... > but the atomic_read_volatile > variant would be more prone to subtle bugs because of the weird > implementation. What bugs? > And it would be more ugly than introducing an order(x) statement for > all memory operations, and adding an order_atomic() wrapper for it > for atomic types. Oh, that order() / forget() macro [forget() was named such by Chuck Ebbert earlier in this thread where he first mentioned it, btw] could definitely be generically introduced for any memory operations. > > a = atomic_read(); > > if (!a) > > do_something(); > > > > forget(); > > a = atomic_read(); > > ... /* some code that depends on value of a, obviously */ > > > > forget(); > > a = atomic_read(); > > ... > > > > So much explicit sprinkling of "forget()" looks ugly. > > Firstly, why is it ugly? It's nice because of those nice explicit > statements there that give us a good heads up and would have some > comments attached to them atomic_read_xxx (where xxx = whatever naming sounds nice to you) would obviously also give a heads up, and could also have some comments attached to it. > (also, lack of the word "volatile" is always a plus). Ok, xxx != volatile. > Secondly, what sort of code would do such a thing? See the nodemgr_host_thread() that does something similar, though not exactly same. > > on the other hand, looks neater. The "_volatile()" suffix makes it also > > no less explicit than an explicit barrier-like macro that this primitive > > is something "special", for code clarity purposes. > > Just don't use the word volatile, That sounds amazingly frivolous, but hey, why not. As I said, ok, xxx != volatile. > and have barriers both before and after the memory operation, How could that lead to bugs? (if you can point to existing code, but just some testcase / sample code would be fine as well). > [...] I don't see > the point though, when you could just have a single barrier(x) > barrier function defined for all memory locations, As I said, barrier() is too heavy-handed. > rather than > this odd thing that only works for atomics Why would it work only for atomics? You could use that generic macro for anything you well damn please. > (and would have to > be duplicated for atomic_set. #define atomic_set_xxx for something similar. Big deal ... NOT. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 10:12 ` Satyam Sharma @ 2007-08-17 12:14 ` Nick Piggin 2007-08-17 13:05 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-17 12:14 UTC (permalink / raw) To: Satyam Sharma Cc: Linus Torvalds, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Satyam Sharma wrote: > >On Fri, 17 Aug 2007, Nick Piggin wrote: > >>I think they would both be equally ugly, >> > >You think both these are equivalent in terms of "looks": > > | >while (!atomic_read(&v)) { | while (!atomic_read_xxx(&v)) { > ... | ... > cpu_relax_no_barrier(); | cpu_relax_no_barrier(); > order_atomic(&v); | } >} | > >(where order_atomic() is an atomic_t >specific wrapper as you mentioned below) > >? > I think the LHS is better if your atomic_read_xxx primitive is using the crazy one-sided barrier, because the LHS code you immediately know what barriers are happening, and with the RHS you have to look at the atomic_read_xxx definition. If your atomic_read_xxx implementation was more intuitive, then both are pretty well equal. More lines != ugly code. >>but the atomic_read_volatile >>variant would be more prone to subtle bugs because of the weird >>implementation. >> > >What bugs? > You can't think for yourself? Your atomic_read_volatile contains a compiler barrier to the atomic variable before the load. 2 such reads from different locations look like this: asm volatile("" : "+m" (v1)); atomic_read(&v1); asm volatile("" : "+m" (v2)); atomic_read(&v2); Which implies that the load of v1 can be reordered to occur after the load of v2. Bet you didn't expect that? >>Secondly, what sort of code would do such a thing? >> > >See the nodemgr_host_thread() that does something similar, though not >exactly same. > I'm sorry, all this waffling about made up code which might do this and that is just a waste of time. Seriously, the thread is bloated enough and never going to get anywhere with all this handwaving. If someone is saving up all the really real and actually good arguments for why we must have a volatile here, now is the time to use them. >>and have barriers both before and after the memory operation, >> > >How could that lead to bugs? (if you can point to existing code, >but just some testcase / sample code would be fine as well). > See above. >As I said, barrier() is too heavy-handed. > Typo. I meant: defined for a single memory location (ie. order(x)). ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 12:14 ` Nick Piggin @ 2007-08-17 13:05 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 13:05 UTC (permalink / raw) To: Nick Piggin Cc: Linus Torvalds, Paul Mackerras, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > [...] > > You think both these are equivalent in terms of "looks": > > > > | > > while (!atomic_read(&v)) { | while (!atomic_read_xxx(&v)) { > > ... | ... > > cpu_relax_no_barrier(); | > > cpu_relax_no_barrier(); > > order_atomic(&v); | } > > } | > > > > (where order_atomic() is an atomic_t > > specific wrapper as you mentioned below) > > > > ? > > I think the LHS is better if your atomic_read_xxx primitive is using the > crazy one-sided barrier, ^^^^^ I'd say it's purposefully one-sided. > because the LHS code you immediately know what > barriers are happening, and with the RHS you have to look at the > atomic_read_xxx > definition. No. As I said, the _xxx (whatever the heck you want to name it as) should give the same heads-up that your "order_atomic" thing is supposed to give. > If your atomic_read_xxx implementation was more intuitive, then both are > pretty well equal. More lines != ugly code. > > > [...] > > What bugs? > > You can't think for yourself? Your atomic_read_volatile contains a compiler > barrier to the atomic variable before the load. 2 such reads from different > locations look like this: > > asm volatile("" : "+m" (v1)); > atomic_read(&v1); > asm volatile("" : "+m" (v2)); > atomic_read(&v2); > > Which implies that the load of v1 can be reordered to occur after the load > of v2. And how would that be a bug? (sorry, I really can't think for myself) > > > Secondly, what sort of code would do such a thing? > > > > See the nodemgr_host_thread() that does something similar, though not > > exactly same. > > I'm sorry, all this waffling about made up code which might do this and > that is just a waste of time. First, you could try looking at the code. And by the way, as I've already said (why do *require* people to have to repeat things to you?) this isn't even about only existing code. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:47 ` Satyam Sharma 2007-08-17 9:15 ` Nick Piggin @ 2007-08-17 9:48 ` Paul Mackerras 2007-08-17 10:23 ` Satyam Sharma 1 sibling, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 9:48 UTC (permalink / raw) To: Satyam Sharma Cc: Nick Piggin, Linus Torvalds, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang Satyam Sharma writes: > I wonder if this'll generate smaller and better code than _both_ the > other atomic_read_volatile() variants. Would need to build allyesconfig > on lots of diff arch's etc to test the theory though. I'm sure it would be a tiny effect. This whole thread is arguing about effects that are quite insignificant. On the one hand we have the non-volatile proponents, who want to let the compiler do extra optimizations - which amounts to letting it elide maybe a dozen loads in the whole kernel, loads which would almost always be L1 cache hits. On the other hand we have the volatile proponents, who are concerned that some code somewhere in the kernel might be buggy without the volatile behaviour, and who also want to be able to remove some barriers and thus save a few bytes of code and a few loads here and there (and possibly some stores too). Either way the effect on code size and execution time is miniscule. In the end the strongest argument is actually that gcc generates unnecessarily verbose code on x86[-64] for volatile accesses. Even then we're only talking about ~2000 bytes, or less than 1 byte per instance of atomic_read on average, about 0.06% of the kernel text size. The x86[-64] developers seem to be willing to bear the debugging cost involved in having the non-volatile behaviour for atomic_read, in order to save the 2kB. That's fine with me. Either way I think somebody should audit all the uses of atomic_read, not just for missing barriers, but also to find the places where it's used in a racy manner. Then we can work out where the races matter and fix them if they do. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 9:48 ` Paul Mackerras @ 2007-08-17 10:23 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 10:23 UTC (permalink / raw) To: Paul Mackerras Cc: Nick Piggin, Linus Torvalds, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Fri, 17 Aug 2007, Paul Mackerras wrote: > Satyam Sharma writes: > > > I wonder if this'll generate smaller and better code than _both_ the > > other atomic_read_volatile() variants. Would need to build allyesconfig > > on lots of diff arch's etc to test the theory though. > > I'm sure it would be a tiny effect. > > This whole thread is arguing about effects that are quite > insignificant. Hmm, the fact that this thread became what it did, probably means that most developers on this list do not mind thinking/arguing about effects or optimizations that are otherwise "tiny". But yeah, they are tiny nonetheless. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 5:56 ` Satyam Sharma 2007-08-17 7:26 ` Nick Piggin @ 2007-08-17 22:49 ` Segher Boessenkool 2007-08-17 23:51 ` Satyam Sharma 1 sibling, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 22:49 UTC (permalink / raw) To: Satyam Sharma Cc: Paul Mackerras, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Nick Piggin, linux-arch, jesper.juhl, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang > #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) > > [ This is exactly equivalent to using "+m" in the constraints, as > recently > explained on a GCC list somewhere, in response to the patch in my > bitops > series a few weeks back where I thought "+m" was bogus. ] [It wasn't explained on a GCC list in response to your patch, as far as I can see -- if I missed it, please point me to an archived version of it]. One last time: it isn't equivalent on older (but still supported by Linux) versions of GCC. Current versions of GCC allow it, but it has no documented behaviour at all, so use it at your own risk. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 22:49 ` Segher Boessenkool @ 2007-08-17 23:51 ` Satyam Sharma 2007-08-17 23:55 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 23:51 UTC (permalink / raw) To: Segher Boessenkool Cc: Paul Mackerras, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Nick Piggin, linux-arch, jesper.juhl, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Sat, 18 Aug 2007, Segher Boessenkool wrote: > > #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) > > > > [ This is exactly equivalent to using "+m" in the constraints, as recently > > explained on a GCC list somewhere, in response to the patch in my bitops > > series a few weeks back where I thought "+m" was bogus. ] > > [It wasn't explained on a GCC list in response to your patch, as > far as I can see -- if I missed it, please point me to an archived > version of it]. http://gcc.gnu.org/ml/gcc-patches/2007-07/msg01758.html is a follow-up in the thread on the gcc-patches@gcc.gnu.org mailing list, which began with: http://gcc.gnu.org/ml/gcc-patches/2007-07/msg01677.html that was posted by Jan Kubicka, as he quotes in that initial posting, after I had submitted: http://lkml.org/lkml/2007/7/23/252 which was a (wrong) patch to "rectify" what I thought was the "bogus" "+m" constraint, as per the quoted extract from gcc docs (that was given in that (wrong) patch's changelog). That's when _I_ came to know how GCC interprets "+m", but probably this has been explained on those lists multiple times. Who cares, anyway? > One last time: it isn't equivalent on older (but still supported > by Linux) versions of GCC. Current versions of GCC allow it, but > it has no documented behaviour at all, so use it at your own risk. True. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 23:51 ` Satyam Sharma @ 2007-08-17 23:55 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 23:55 UTC (permalink / raw) To: Satyam Sharma Cc: Paul Mackerras, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, ak, netdev, cfriesen, Nick Piggin, linux-arch, jesper.juhl, Andrew Morton, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >>> #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) >>> >>> [ This is exactly equivalent to using "+m" in the constraints, as >>> recently >>> explained on a GCC list somewhere, in response to the patch in my >>> bitops >>> series a few weeks back where I thought "+m" was bogus. ] >> >> [It wasn't explained on a GCC list in response to your patch, as >> far as I can see -- if I missed it, please point me to an archived >> version of it]. > > http://gcc.gnu.org/ml/gcc-patches/2007-07/msg01758.html Ah yes, that old thread, thank you. > That's when _I_ came to know how GCC interprets "+m", but probably > this has been explained on those lists multiple times. Who cares, > anyway? I just couldn't find the thread you meant, I thought I missed have it, that's all :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:42 ` Linus Torvalds 2007-08-17 5:18 ` Paul E. McKenney 2007-08-17 5:56 ` Satyam Sharma @ 2007-08-17 6:42 ` Geert Uytterhoeven 2007-08-17 8:52 ` Andi Kleen 2007-08-17 22:29 ` Segher Boessenkool 4 siblings, 0 replies; 333+ messages in thread From: Geert Uytterhoeven @ 2007-08-17 6:42 UTC (permalink / raw) To: Linus Torvalds Cc: Paul Mackerras, Nick Piggin, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Thu, 16 Aug 2007, Linus Torvalds wrote: > On Fri, 17 Aug 2007, Paul Mackerras wrote: > > I'm really surprised it's as much as a few K. I tried it on powerpc > > and it only saved 40 bytes (10 instructions) for a G5 config. > > One of the things that "volatile" generally screws up is a simple > > volatile int i; > > i++; > > which a compiler will generally get horribly, horribly wrong. > > In a reasonable world, gcc should just make that be (on x86) > > addl $1,i(%rip) > > on x86-64, which is indeed what it does without the volatile. But with the > volatile, the compiler gets really nervous, and doesn't dare do it in one > instruction, and thus generates crap like > > movl i(%rip), %eax > addl $1, %eax > movl %eax, i(%rip) > > instead. For no good reason, except that "volatile" just doesn't have any > good/clear semantics for the compiler, so most compilers will just make it > be "I will not touch this access in any way, shape, or form". Including > even trivially correct instruction optimization/combination. Apart from having to fetch more bytes for the instructions (which does matter), execution time is probably the same on modern processors, as they convert the single instruction to RISC-style load, modify, store anyway. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:42 ` Linus Torvalds ` (2 preceding siblings ...) 2007-08-17 6:42 ` Geert Uytterhoeven @ 2007-08-17 8:52 ` Andi Kleen 2007-08-17 10:08 ` Satyam Sharma 2007-08-17 22:29 ` Segher Boessenkool 4 siblings, 1 reply; 333+ messages in thread From: Andi Kleen @ 2007-08-17 8:52 UTC (permalink / raw) To: Linus Torvalds Cc: Paul Mackerras, Nick Piggin, Segher Boessenkool, heiko.carstens, horms, linux-kernel, rpjday, netdev, cfriesen, akpm, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Friday 17 August 2007 05:42, Linus Torvalds wrote: > On Fri, 17 Aug 2007, Paul Mackerras wrote: > > I'm really surprised it's as much as a few K. I tried it on powerpc > > and it only saved 40 bytes (10 instructions) for a G5 config. > > One of the things that "volatile" generally screws up is a simple > > volatile int i; > > i++; But for atomic_t people use atomic_inc() anyways which does this correctly. It shouldn't really matter for atomic_t. I'm worrying a bit that the volatile atomic_t change caused subtle code breakage like these delay read loops people here pointed out. Wouldn't it be safer to just re-add the volatile to atomic_read() for 2.6.23? Or alternatively make it asm(), but volatile seems more proven. -Andi ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:52 ` Andi Kleen @ 2007-08-17 10:08 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 10:08 UTC (permalink / raw) To: Andi Kleen Cc: Linus Torvalds, Paul Mackerras, Nick Piggin, Segher Boessenkool, heiko.carstens, horms, Linux Kernel Mailing List, rpjday, netdev, cfriesen, Andrew Morton, jesper.juhl, linux-arch, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang On Fri, 17 Aug 2007, Andi Kleen wrote: > On Friday 17 August 2007 05:42, Linus Torvalds wrote: > > On Fri, 17 Aug 2007, Paul Mackerras wrote: > > > I'm really surprised it's as much as a few K. I tried it on powerpc > > > and it only saved 40 bytes (10 instructions) for a G5 config. > > > > One of the things that "volatile" generally screws up is a simple > > > > volatile int i; > > > > i++; > > But for atomic_t people use atomic_inc() anyways which does this correctly. > It shouldn't really matter for atomic_t. > > I'm worrying a bit that the volatile atomic_t change caused subtle code > breakage like these delay read loops people here pointed out. Umm, I followed most of the thread, but which breakage is this? > Wouldn't it be safer to just re-add the volatile to atomic_read() > for 2.6.23? Or alternatively make it asm(), but volatile seems more > proven. The problem with volatile is not just trashy code generation (which also definitely is a major problem), but definition holes, and implementation inconsistencies. Making it asm() is not the only other alternative to volatile either (read another reply to this mail), but considering most of the thread has been about people not wanting even a atomic_read_volatile() variant, making atomic_read() itself have volatile semantics sounds ... strange :-) PS: http://lkml.org/lkml/2007/8/15/407 was submitted a couple days back, any word if you saw that? I have another one for you: [PATCH] i386, x86_64: __const_udelay() should not be marked inline Because it can never get inlined in any callsite (each translation unit is compiled separately for the kernel and so the implementation of __const_udelay() would be invisible to all other callsites). In fact it turns out, the correctness of callsites at arch/x86_64/kernel/crash.c:97 and arch/i386/kernel/crash.c:101 explicitly _depends_ upon it not being inlined, and also it's an exported symbol (modules may want to call mdelay() and udelay() that often becomes __const_udelay() after some macro-ing in various headers). So let's not mark it as "inline" either. Signed-off-by: Satyam Sharma <satyam@infradead.org> --- arch/i386/lib/delay.c | 2 +- arch/x86_64/lib/delay.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/i386/lib/delay.c b/arch/i386/lib/delay.c index f6edb11..0082c99 100644 --- a/arch/i386/lib/delay.c +++ b/arch/i386/lib/delay.c @@ -74,7 +74,7 @@ void __delay(unsigned long loops) delay_fn(loops); } -inline void __const_udelay(unsigned long xloops) +void __const_udelay(unsigned long xloops) { int d0; diff --git a/arch/x86_64/lib/delay.c b/arch/x86_64/lib/delay.c index 2dbebd3..d0cd9cd 100644 --- a/arch/x86_64/lib/delay.c +++ b/arch/x86_64/lib/delay.c @@ -38,7 +38,7 @@ void __delay(unsigned long loops) } EXPORT_SYMBOL(__delay); -inline void __const_udelay(unsigned long xloops) +void __const_udelay(unsigned long xloops) { __delay(((xloops * HZ * cpu_data[raw_smp_processor_id()].loops_per_jiffy) >> 32) + 1); } ^ permalink raw reply related [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:42 ` Linus Torvalds ` (3 preceding siblings ...) 2007-08-17 8:52 ` Andi Kleen @ 2007-08-17 22:29 ` Segher Boessenkool 4 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 22:29 UTC (permalink / raw) To: Linus Torvalds Cc: Paul Mackerras, heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, Nick Piggin, linux-arch, jesper.juhl, satyam, zlynx, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang > In a reasonable world, gcc should just make that be (on x86) > > addl $1,i(%rip) > > on x86-64, which is indeed what it does without the volatile. But with > the > volatile, the compiler gets really nervous, and doesn't dare do it in > one > instruction, and thus generates crap like > > movl i(%rip), %eax > addl $1, %eax > movl %eax, i(%rip) > > instead. For no good reason, except that "volatile" just doesn't have > any > good/clear semantics for the compiler, so most compilers will just > make it > be "I will not touch this access in any way, shape, or form". Including > even trivially correct instruction optimization/combination. It's just a (target-specific, perhaps) missed-optimisation kind of bug in GCC. Care to file a bug report? > but is > (again) something that gcc doesn't dare do, since "i" is volatile. Just nobody taught it it can do this; perhaps no one wanted to add optimisations like that, maybe with a reasoning like "people who hit the go-slow-in-unspecified-ways button should get what they deserve" ;-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 2:19 ` Nick Piggin 2007-08-17 3:16 ` Paul Mackerras @ 2007-08-17 17:37 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 17:37 UTC (permalink / raw) To: Nick Piggin Cc: heiko.carstens, horms, linux-kernel, rpjday, ak, netdev, cfriesen, akpm, torvalds, jesper.juhl, linux-arch, zlynx, satyam, clameter, schwidefsky, Chris Snook, Herbert Xu, davem, wensong, wjiang >>>>>> Part of the motivation here is to fix heisenbugs. If I knew >>>>>> where they >>>>> >>>>> By the same token we should probably disable optimisations >>>>> altogether since that too can create heisenbugs. >>>> >>>> Almost everything is a tradeoff; and so is this. I don't >>>> believe most people would find disabling all compiler >>>> optimisations an acceptable price to pay for some peace >>>> of mind. >>> >>> >>> So why is this a good tradeoff? >> It certainly is better than disabling all compiler optimisations! > > It's easy to be better than something really stupid :) Sure, it wasn't me who made the comparison though. > So i386 and x86-64 don't have volatiles there, and it saves them a > few K of kernel text. Which has to be investigated. A few kB is a lot more than expected. > What you need to justify is why it is a good > tradeoff to make them volatile (which btw, is much harder to go > the other way after we let people make those assumptions). My point is that people *already* made those assumptions. There are two ways to clean up this mess: 1) Have the "volatile" semantics by default, change the users that don't need it; 2) Have "non-volatile" semantics by default, change the users that do need it. Option 2) randomly breaks stuff all over the place, option 1) doesn't. Yeah 1) could cause some extremely minor speed or code size regression, but only temporarily until everything has been audited. >>> I also think that just adding things to APIs in the hope it might fix >>> up some bugs isn't really a good road to go down. Where do you stop? >> I look at it the other way: keeping the "volatile" semantics in >> atomic_XXX() (or adding them to it, whatever) helps _prevent_ bugs; > > Yeah, but we could add lots of things to help prevent bugs and > would never be included. I would also contend that it helps _hide_ > bugs and encourages people to be lazy when thinking about these > things. Sure. We aren't _adding_ anything here though, not on the platforms where it is most likely to show up, anyway. > Also, you dismiss the fact that we'd actually be *adding* volatile > semantics back to the 2 most widely tested architectures (in terms > of test time, number of testers, variety of configurations, and > coverage of driver code). I'm not dismissing that. x86 however is one of the few architectures where mistakenly leaving out a "volatile" will not easily show up on user testing, since the compiler will very often produce a memory reference anyway because it has no registers to play with. > This is a very important different from > just keeping volatile semantics because it is basically a one-way > API change. That's a good point. Maybe we should create _two_ new APIs, one explicitly going each way. >> certainly most people expect that behaviour, and also that behaviour >> is *needed* in some places and no other interface provides that >> functionality. > > I don't know that most people would expect that behaviour. I didn't conduct any formal poll either :-) > Is there any documentation anywhere that would suggest this? Not really I think, no. But not the other way around, either. Most uses of it seem to expect it though. >> [some confusion about barriers wrt atomics snipped] > > What were you confused about? Me? Not much. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 23:08 ` Satyam Sharma 2007-08-14 23:04 ` Chris Snook @ 2007-08-14 23:26 ` Paul E. McKenney 2007-08-15 10:35 ` Stefan Richter 2 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-14 23:26 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 04:38:54AM +0530, Satyam Sharma wrote: > > > On Tue, 14 Aug 2007, Christoph Lameter wrote: > > > On Thu, 9 Aug 2007, Chris Snook wrote: > > > > > This patchset makes the behavior of atomic_read uniform by removing the > > > volatile keyword from all atomic_t and atomic64_t definitions that currently > > > have it, and instead explicitly casts the variable as volatile in > > > atomic_read(). This leaves little room for creative optimization by the > > > compiler, and is in keeping with the principles behind "volatile considered > > > harmful". > > > > volatile is generally harmful even in atomic_read(). Barriers control > > visibility and AFAICT things are fine. > > Frankly, I don't see the need for this series myself either. Personal > opinion (others may differ), but I consider "volatile" to be a sad / > unfortunate wart in C (numerous threads on this list and on the gcc > lists/bugzilla over the years stand testimony to this) and if we _can_ > steer clear of it, then why not -- why use this ill-defined primitive > whose implementation has often differed over compiler versions and > platforms? Granted, barrier() _is_ heavy-handed in that it makes the > optimizer forget _everything_, but then somebody did post a forget() > macro on this thread itself ... > > [ BTW, why do we want the compiler to not optimize atomic_read()'s in > the first place? Atomic ops guarantee atomicity, which has nothing > to do with "volatility" -- users that expect "volatility" from > atomic ops are the ones who must be fixed instead, IMHO. ] Interactions between mainline code and interrupt/NMI handlers on the same CPU (for example, when both are using per-CPU variables. See examples previously posted in this thread, or look at the rcu_read_lock() and rcu_read_unlock() implementations in http://lkml.org/lkml/2007/8/7/280. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-14 23:08 ` Satyam Sharma 2007-08-14 23:04 ` Chris Snook 2007-08-14 23:26 ` Paul E. McKenney @ 2007-08-15 10:35 ` Stefan Richter 2007-08-15 12:04 ` Herbert Xu ` (2 more replies) 2 siblings, 3 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-15 10:35 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney Satyam Sharma wrote: > [ BTW, why do we want the compiler to not optimize atomic_read()'s in > the first place? Atomic ops guarantee atomicity, which has nothing > to do with "volatility" -- users that expect "volatility" from > atomic ops are the ones who must be fixed instead, IMHO. ] LDD3 says on page 125: "The following operations are defined for the type [atomic_t] and are guaranteed to be atomic with respect to all processors of an SMP computer." Doesn't "atomic WRT all processors" require volatility? -- Stefan Richter -=====-=-=== =--- -==== http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 10:35 ` Stefan Richter @ 2007-08-15 12:04 ` Herbert Xu 2007-08-15 12:31 ` Satyam Sharma 2007-08-15 19:59 ` Christoph Lameter 2 siblings, 0 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-15 12:04 UTC (permalink / raw) To: Stefan Richter Cc: Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Paul E. McKenney On Wed, Aug 15, 2007 at 12:35:31PM +0200, Stefan Richter wrote: > > LDD3 says on page 125: "The following operations are defined for the > type [atomic_t] and are guaranteed to be atomic with respect to all > processors of an SMP computer." > > Doesn't "atomic WRT all processors" require volatility? Not at all. We also require this to be atomic without any hint of volatility. extern int foo; foo = 4; Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 10:35 ` Stefan Richter 2007-08-15 12:04 ` Herbert Xu @ 2007-08-15 12:31 ` Satyam Sharma 2007-08-15 13:08 ` Stefan Richter ` (3 more replies) 2007-08-15 19:59 ` Christoph Lameter 2 siblings, 4 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 12:31 UTC (permalink / raw) To: Stefan Richter Cc: Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Wed, 15 Aug 2007, Stefan Richter wrote: > Satyam Sharma wrote: > > [ BTW, why do we want the compiler to not optimize atomic_read()'s in > > the first place? Atomic ops guarantee atomicity, which has nothing > > to do with "volatility" -- users that expect "volatility" from > > atomic ops are the ones who must be fixed instead, IMHO. ] > > LDD3 says on page 125: "The following operations are defined for the > type [atomic_t] and are guaranteed to be atomic with respect to all > processors of an SMP computer." > > Doesn't "atomic WRT all processors" require volatility? No, it definitely doesn't. Why should it? "Atomic w.r.t. all processors" is just your normal, simple "atomicity" for SMP systems (ensure that that object is modified / set / replaced in main memory atomically) and has nothing to do with "volatile" behaviour. "Volatile behaviour" itself isn't consistently defined (at least definitely not consistently implemented in various gcc versions across platforms), but it is /expected/ to mean something like: "ensure that every such access actually goes all the way to memory, and is not re-ordered w.r.t. to other accesses, as far as the compiler can take care of these". The last "as far as compiler can take care" disclaimer comes about due to CPUs doing their own re-ordering nowadays. For example (say on i386): (A) $ cat tp1.c int a; void func(void) { a = 10; a = 20; } $ gcc -Os -S tp1.c $ cat tp1.s ... movl $20, a ... (B) $ cat tp2.c volatile int a; void func(void) { a = 10; a = 20; } $ gcc -Os -S tp2.c $ cat tp2.s ... movl $10, a movl $20, a ... (C) $ cat tp3.c int a; void func(void) { *(volatile int *)&a = 10; *(volatile int *)&a = 20; } $ gcc -Os -S tp3.c $ cat tp3.s ... movl $10, a movl $20, a ... In (A) the compiler optimized "a = 10;" away, but the actual store of the final value "20" to "a" was still "atomic". (B) and (C) also exhibit "volatile" behaviour apart from the "atomicity". But as others replied, it seems some callers out there depend upon atomic ops exhibiting "volatile" behaviour as well, so that answers my initial question, actually. I haven't looked at the code Paul pointed me at, but I wonder if that "forget(x)" macro would help those cases. I'd wish to avoid the "volatile" primitive, personally. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 12:31 ` Satyam Sharma @ 2007-08-15 13:08 ` Stefan Richter 2007-08-15 13:11 ` Stefan Richter 2007-08-15 13:47 ` Satyam Sharma 2007-08-15 18:31 ` Segher Boessenkool ` (2 subsequent siblings) 3 siblings, 2 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-15 13:08 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney Satyam Sharma wrote: > On Wed, 15 Aug 2007, Stefan Richter wrote: >> Doesn't "atomic WRT all processors" require volatility? > > No, it definitely doesn't. Why should it? > > "Atomic w.r.t. all processors" is just your normal, simple "atomicity" > for SMP systems (ensure that that object is modified / set / replaced > in main memory atomically) and has nothing to do with "volatile" > behaviour. > > "Volatile behaviour" itself isn't consistently defined (at least > definitely not consistently implemented in various gcc versions across > platforms), but it is /expected/ to mean something like: "ensure that > every such access actually goes all the way to memory, and is not > re-ordered w.r.t. to other accesses, as far as the compiler can take > care of these". The last "as far as compiler can take care" disclaimer > comes about due to CPUs doing their own re-ordering nowadays. > > For example (say on i386): [...] > In (A) the compiler optimized "a = 10;" away, but the actual store > of the final value "20" to "a" was still "atomic". (B) and (C) also > exhibit "volatile" behaviour apart from the "atomicity". > > But as others replied, it seems some callers out there depend upon > atomic ops exhibiting "volatile" behaviour as well, so that answers > my initial question, actually. I haven't looked at the code Paul > pointed me at, but I wonder if that "forget(x)" macro would help > those cases. I'd wish to avoid the "volatile" primitive, personally. So, looking at load instead of store, understand I correctly that in your opinion int b; b = atomic_read(&a); if (b) do_something_time_consuming(); b = atomic_read(&a); if (b) do_something_more(); should be changed to explicitly forget(&a) after do_something_time_consuming? If so, how about the following: static inline void A(atomic_t *a) { int b = atomic_read(a); if (b) do_something_time_consuming(); } static inline void B(atomic_t *a) { int b = atomic_read(a); if (b) do_something_more(); } static void C(atomic_t *a) { A(a); B(b); } Would this need forget(a) after A(a)? (Is the latter actually answered in C99 or is it compiler-dependent?) -- Stefan Richter -=====-=-=== =--- -==== http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 13:08 ` Stefan Richter @ 2007-08-15 13:11 ` Stefan Richter 2007-08-15 13:47 ` Satyam Sharma 1 sibling, 0 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-15 13:11 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney I wrote: > static inline void A(atomic_t *a) > { > int b = atomic_read(a); > if (b) > do_something_time_consuming(); > } > > static inline void B(atomic_t *a) > { > int b = atomic_read(a); > if (b) > do_something_more(); > } > > static void C(atomic_t *a) > { > A(a); > B(b); /* ^ typo */ B(a); > } > > Would this need forget(a) after A(a)? > > (Is the latter actually answered in C99 or is it compiler-dependent?) -- Stefan Richter -=====-=-=== =--- -==== http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 13:08 ` Stefan Richter 2007-08-15 13:11 ` Stefan Richter @ 2007-08-15 13:47 ` Satyam Sharma 2007-08-15 14:25 ` Paul E. McKenney 1 sibling, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 13:47 UTC (permalink / raw) To: Stefan Richter Cc: Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Wed, 15 Aug 2007, Stefan Richter wrote: > Satyam Sharma wrote: > > On Wed, 15 Aug 2007, Stefan Richter wrote: > >> Doesn't "atomic WRT all processors" require volatility? > > > > No, it definitely doesn't. Why should it? > > > > "Atomic w.r.t. all processors" is just your normal, simple "atomicity" > > for SMP systems (ensure that that object is modified / set / replaced > > in main memory atomically) and has nothing to do with "volatile" > > behaviour. > > > > "Volatile behaviour" itself isn't consistently defined (at least > > definitely not consistently implemented in various gcc versions across > > platforms), but it is /expected/ to mean something like: "ensure that > > every such access actually goes all the way to memory, and is not > > re-ordered w.r.t. to other accesses, as far as the compiler can take > > care of these". The last "as far as compiler can take care" disclaimer > > comes about due to CPUs doing their own re-ordering nowadays. > > > > For example (say on i386): > > [...] > > > In (A) the compiler optimized "a = 10;" away, but the actual store > > of the final value "20" to "a" was still "atomic". (B) and (C) also > > exhibit "volatile" behaviour apart from the "atomicity". > > > > But as others replied, it seems some callers out there depend upon > > atomic ops exhibiting "volatile" behaviour as well, so that answers > > my initial question, actually. I haven't looked at the code Paul > > pointed me at, but I wonder if that "forget(x)" macro would help > > those cases. I'd wish to avoid the "volatile" primitive, personally. > > So, looking at load instead of store, understand I correctly that in > your opinion > > int b; > > b = atomic_read(&a); > if (b) > do_something_time_consuming(); > > b = atomic_read(&a); > if (b) > do_something_more(); > > should be changed to explicitly forget(&a) after > do_something_time_consuming? No, I'd actually prefer something like what Christoph Lameter suggested, i.e. users (such as above) who want "volatile"-like behaviour from atomic ops can use alternative functions. How about something like: #define atomic_read_volatile(v) \ ({ \ forget(&(v)->counter); \ ((v)->counter); \ }) Or possibly, implement these "volatile" atomic ops variants in inline asm like the patch that Sebastian Siewior has submitted on another thread just a while back. Of course, if we find there are more callers in the kernel who want the volatility behaviour than those who don't care, we can re-define the existing ops to such variants, and re-name the existing definitions to somethine else, say "atomic_read_nonvolatile" for all I care. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 13:47 ` Satyam Sharma @ 2007-08-15 14:25 ` Paul E. McKenney 2007-08-15 15:33 ` Herbert Xu ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 14:25 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, Aug 15, 2007 at 07:17:29PM +0530, Satyam Sharma wrote: > On Wed, 15 Aug 2007, Stefan Richter wrote: > > Satyam Sharma wrote: > > > On Wed, 15 Aug 2007, Stefan Richter wrote: > > >> Doesn't "atomic WRT all processors" require volatility? > > > > > > No, it definitely doesn't. Why should it? > > > > > > "Atomic w.r.t. all processors" is just your normal, simple "atomicity" > > > for SMP systems (ensure that that object is modified / set / replaced > > > in main memory atomically) and has nothing to do with "volatile" > > > behaviour. > > > > > > "Volatile behaviour" itself isn't consistently defined (at least > > > definitely not consistently implemented in various gcc versions across > > > platforms), but it is /expected/ to mean something like: "ensure that > > > every such access actually goes all the way to memory, and is not > > > re-ordered w.r.t. to other accesses, as far as the compiler can take > > > care of these". The last "as far as compiler can take care" disclaimer > > > comes about due to CPUs doing their own re-ordering nowadays. > > > > > > For example (say on i386): > > > > [...] > > > > > In (A) the compiler optimized "a = 10;" away, but the actual store > > > of the final value "20" to "a" was still "atomic". (B) and (C) also > > > exhibit "volatile" behaviour apart from the "atomicity". > > > > > > But as others replied, it seems some callers out there depend upon > > > atomic ops exhibiting "volatile" behaviour as well, so that answers > > > my initial question, actually. I haven't looked at the code Paul > > > pointed me at, but I wonder if that "forget(x)" macro would help > > > those cases. I'd wish to avoid the "volatile" primitive, personally. > > > > So, looking at load instead of store, understand I correctly that in > > your opinion > > > > int b; > > > > b = atomic_read(&a); > > if (b) > > do_something_time_consuming(); > > > > b = atomic_read(&a); > > if (b) > > do_something_more(); > > > > should be changed to explicitly forget(&a) after > > do_something_time_consuming? > > No, I'd actually prefer something like what Christoph Lameter suggested, > i.e. users (such as above) who want "volatile"-like behaviour from atomic > ops can use alternative functions. How about something like: > > #define atomic_read_volatile(v) \ > ({ \ > forget(&(v)->counter); \ > ((v)->counter); \ > }) Wouldn't the above "forget" the value, throw it away, then forget that it forgot it, giving non-volatile semantics? > Or possibly, implement these "volatile" atomic ops variants in inline asm > like the patch that Sebastian Siewior has submitted on another thread just > a while back. Given that you are advocating a change (please keep in mind that atomic_read() and atomic_set() had volatile semantics on almost all platforms), care to give some example where these historical volatile semantics are causing a problem? > Of course, if we find there are more callers in the kernel who want the > volatility behaviour than those who don't care, we can re-define the > existing ops to such variants, and re-name the existing definitions to > somethine else, say "atomic_read_nonvolatile" for all I care. Do we really need another set of APIs? Can you give even one example where the pre-existing volatile semantics are causing enough of a problem to justify adding yet more atomic_*() APIs? Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 14:25 ` Paul E. McKenney @ 2007-08-15 15:33 ` Herbert Xu 2007-08-15 16:08 ` Paul E. McKenney 2007-08-15 17:55 ` Satyam Sharma 2007-08-15 18:19 ` David Howells 2 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-15 15:33 UTC (permalink / raw) To: Paul E. McKenney Cc: Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote: > > Do we really need another set of APIs? Can you give even one example > where the pre-existing volatile semantics are causing enough of a problem > to justify adding yet more atomic_*() APIs? Let's turn this around. Can you give a single example where the volatile semantics is needed in a legitimate way? Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 15:33 ` Herbert Xu @ 2007-08-15 16:08 ` Paul E. McKenney 2007-08-15 17:18 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 16:08 UTC (permalink / raw) To: Herbert Xu Cc: Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 11:33:36PM +0800, Herbert Xu wrote: > On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote: > > > > Do we really need another set of APIs? Can you give even one example > > where the pre-existing volatile semantics are causing enough of a problem > > to justify adding yet more atomic_*() APIs? > > Let's turn this around. Can you give a single example where > the volatile semantics is needed in a legitimate way? Sorry, but you are the one advocating for the change. Nice try, though! ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 16:08 ` Paul E. McKenney @ 2007-08-15 17:18 ` Satyam Sharma 2007-08-15 17:33 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 17:18 UTC (permalink / raw) To: Paul E. McKenney Cc: Herbert Xu, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, 15 Aug 2007, Paul E. McKenney wrote: > On Wed, Aug 15, 2007 at 11:33:36PM +0800, Herbert Xu wrote: > > On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote: > > > > > > Do we really need another set of APIs? Can you give even one example > > > where the pre-existing volatile semantics are causing enough of a problem > > > to justify adding yet more atomic_*() APIs? > > > > Let's turn this around. Can you give a single example where > > the volatile semantics is needed in a legitimate way? > > Sorry, but you are the one advocating for the change. Not for i386 and x86_64 -- those have atomic ops without any "volatile" semantics (currently as per existing definitions). ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 17:18 ` Satyam Sharma @ 2007-08-15 17:33 ` Paul E. McKenney 2007-08-15 18:05 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 17:33 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 10:48:28PM +0530, Satyam Sharma wrote: > On Wed, 15 Aug 2007, Paul E. McKenney wrote: > > On Wed, Aug 15, 2007 at 11:33:36PM +0800, Herbert Xu wrote: > > > On Wed, Aug 15, 2007 at 07:25:16AM -0700, Paul E. McKenney wrote: > > > > > > > > Do we really need another set of APIs? Can you give even one example > > > > where the pre-existing volatile semantics are causing enough of a problem > > > > to justify adding yet more atomic_*() APIs? > > > > > > Let's turn this around. Can you give a single example where > > > the volatile semantics is needed in a legitimate way? > > > > Sorry, but you are the one advocating for the change. > > Not for i386 and x86_64 -- those have atomic ops without any "volatile" > semantics (currently as per existing definitions). I claim unit volumes with arm, and the majority of the architectures, but I cannot deny the popularity of i386 and x86_64 with many developers. ;-) However, I am not aware of code in the kernel that would benefit from the compiler coalescing multiple atomic_set() and atomic_read() invocations, thus I don't see the downside to volatility in this case. Are there some performance-critical code fragments that I am missing? Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 17:33 ` Paul E. McKenney @ 2007-08-15 18:05 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 18:05 UTC (permalink / raw) To: Paul E. McKenney Cc: Herbert Xu, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, 15 Aug 2007, Paul E. McKenney wrote: > On Wed, Aug 15, 2007 at 10:48:28PM +0530, Satyam Sharma wrote: > > [...] > > Not for i386 and x86_64 -- those have atomic ops without any "volatile" > > semantics (currently as per existing definitions). > > I claim unit volumes with arm, and the majority of the architectures, but > I cannot deny the popularity of i386 and x86_64 with many developers. ;-) Hmm, does arm really need that "volatile int counter;"? Hopefully RMK will take a patch removing that "volatile" ... ;-) > However, I am not aware of code in the kernel that would benefit > from the compiler coalescing multiple atomic_set() and atomic_read() > invocations, thus I don't see the downside to volatility in this case. > Are there some performance-critical code fragments that I am missing? I don't know, and yes, code with multiple atomic_set's and atomic_read's getting optimized or coalesced does sound strange to start with. Anyway, I'm not against "volatile semantics" per se. As replied elsewhere, I do appreciate the motivation behind this series (to _avoid_ gotchas, not to fix existing ones). Just that I'd like to avoid using "volatile", for aforementioned reasons, especially given that there are perfectly reasonable alternatives to achieve the same desired behaviour. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 14:25 ` Paul E. McKenney 2007-08-15 15:33 ` Herbert Xu @ 2007-08-15 17:55 ` Satyam Sharma 2007-08-15 19:07 ` Paul E. McKenney 2007-08-15 20:58 ` Segher Boessenkool 2007-08-15 18:19 ` David Howells 2 siblings, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 17:55 UTC (permalink / raw) To: Paul E. McKenney Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu Hi Paul, On Wed, 15 Aug 2007, Paul E. McKenney wrote: > On Wed, Aug 15, 2007 at 07:17:29PM +0530, Satyam Sharma wrote: > > [...] > > No, I'd actually prefer something like what Christoph Lameter suggested, > > i.e. users (such as above) who want "volatile"-like behaviour from atomic > > ops can use alternative functions. How about something like: > > > > #define atomic_read_volatile(v) \ > > ({ \ > > forget(&(v)->counter); \ > > ((v)->counter); \ > > }) > > Wouldn't the above "forget" the value, throw it away, then forget > that it forgot it, giving non-volatile semantics? Nope, I don't think so. I wrote the following trivial testcases: [ See especially tp4.c and tp4.s (last example). ] ============================================================================== $ cat tp1.c # Using volatile access casts #define atomic_read(a) (*(volatile int *)&a) int a; void func(void) { a = 0; while (atomic_read(a)) ; } ============================================================================== $ gcc -Os -S tp1.c; cat tp1.s func: pushl %ebp movl %esp, %ebp movl $0, a .L2: movl a, %eax testl %eax, %eax jne .L2 popl %ebp ret ============================================================================== $ cat tp2.c # Using nothing; gcc will optimize the whole loop away #define forget(x) #define atomic_read(a) \ ({ \ forget(&(a)); \ (a); \ }) int a; void func(void) { a = 0; while (atomic_read(a)) ; } ============================================================================== $ gcc -Os -S tp2.c; cat tp2.s func: pushl %ebp movl %esp, %ebp popl %ebp movl $0, a ret ============================================================================== $ cat tp3.c # Using a full memory clobber barrier #define forget(x) asm volatile ("":::"memory") #define atomic_read(a) \ ({ \ forget(&(a)); \ (a); \ }) int a; void func(void) { a = 0; while (atomic_read(a)) ; } ============================================================================== $ gcc -Os -S tp3.c; cat tp3.s func: pushl %ebp movl %esp, %ebp movl $0, a .L2: cmpl $0, a jne .L2 popl %ebp ret ============================================================================== $ cat tp4.c # Using a forget(var) macro #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) #define atomic_read(a) \ ({ \ forget(a); \ (a); \ }) int a; void func(void) { a = 0; while (atomic_read(a)) ; } ============================================================================== $ gcc -Os -S tp4.c; cat tp4.s func: pushl %ebp movl %esp, %ebp movl $0, a .L2: cmpl $0, a jne .L2 popl %ebp ret ============================================================================== Possibly these were too trivial to expose any potential problems that you may have been referring to, so would be helpful if you could write a more concrete example / sample code. > > Or possibly, implement these "volatile" atomic ops variants in inline asm > > like the patch that Sebastian Siewior has submitted on another thread just > > a while back. > > Given that you are advocating a change (please keep in mind that > atomic_read() and atomic_set() had volatile semantics on almost all > platforms), care to give some example where these historical volatile > semantics are causing a problem? > [...] > Can you give even one example > where the pre-existing volatile semantics are causing enough of a problem > to justify adding yet more atomic_*() APIs? Will take this to the other sub-thread ... > > Of course, if we find there are more callers in the kernel who want the > > volatility behaviour than those who don't care, we can re-define the > > existing ops to such variants, and re-name the existing definitions to > > somethine else, say "atomic_read_nonvolatile" for all I care. > > Do we really need another set of APIs? Well, if there's one set of users who do care about volatile behaviour, and another set that doesn't, it only sounds correct to provide both those APIs, instead of forcing one behaviour on all users. Thanks, Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 17:55 ` Satyam Sharma @ 2007-08-15 19:07 ` Paul E. McKenney 2007-08-15 21:07 ` Segher Boessenkool 2007-08-15 20:58 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 19:07 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, Aug 15, 2007 at 11:25:05PM +0530, Satyam Sharma wrote: > Hi Paul, > On Wed, 15 Aug 2007, Paul E. McKenney wrote: > > > On Wed, Aug 15, 2007 at 07:17:29PM +0530, Satyam Sharma wrote: > > > [...] > > > No, I'd actually prefer something like what Christoph Lameter suggested, > > > i.e. users (such as above) who want "volatile"-like behaviour from atomic > > > ops can use alternative functions. How about something like: > > > > > > #define atomic_read_volatile(v) \ > > > ({ \ > > > forget(&(v)->counter); \ > > > ((v)->counter); \ > > > }) > > > > Wouldn't the above "forget" the value, throw it away, then forget > > that it forgot it, giving non-volatile semantics? > > Nope, I don't think so. I wrote the following trivial testcases: > [ See especially tp4.c and tp4.s (last example). ] Right. I should have said "wouldn't the compiler be within its rights to forget the value, throw it away, then forget that it forgot it". The value coming out of the #define above is an unadorned ((v)->counter), which has no volatile semantics. > ============================================================================== > $ cat tp1.c # Using volatile access casts > > #define atomic_read(a) (*(volatile int *)&a) > > int a; > > void func(void) > { > a = 0; > while (atomic_read(a)) > ; > } > ============================================================================== > $ gcc -Os -S tp1.c; cat tp1.s > > func: > pushl %ebp > movl %esp, %ebp > movl $0, a > .L2: > movl a, %eax > testl %eax, %eax > jne .L2 > popl %ebp > ret > ============================================================================== > $ cat tp2.c # Using nothing; gcc will optimize the whole loop away > > #define forget(x) > > #define atomic_read(a) \ > ({ \ > forget(&(a)); \ > (a); \ > }) > > int a; > > void func(void) > { > a = 0; > while (atomic_read(a)) > ; > } > ============================================================================== > $ gcc -Os -S tp2.c; cat tp2.s > > func: > pushl %ebp > movl %esp, %ebp > popl %ebp > movl $0, a > ret > ============================================================================== > $ cat tp3.c # Using a full memory clobber barrier > > #define forget(x) asm volatile ("":::"memory") > > #define atomic_read(a) \ > ({ \ > forget(&(a)); \ > (a); \ > }) > > int a; > > void func(void) > { > a = 0; > while (atomic_read(a)) > ; > } > ============================================================================== > $ gcc -Os -S tp3.c; cat tp3.s > > func: > pushl %ebp > movl %esp, %ebp > movl $0, a > .L2: > cmpl $0, a > jne .L2 > popl %ebp > ret > ============================================================================== > $ cat tp4.c # Using a forget(var) macro > > #define forget(a) __asm__ __volatile__ ("" :"=m" (a) :"m" (a)) > > #define atomic_read(a) \ > ({ \ > forget(a); \ > (a); \ > }) > > int a; > > void func(void) > { > a = 0; > while (atomic_read(a)) > ; > } > ============================================================================== > $ gcc -Os -S tp4.c; cat tp4.s > > func: > pushl %ebp > movl %esp, %ebp > movl $0, a > .L2: > cmpl $0, a > jne .L2 > popl %ebp > ret > ============================================================================== > > Possibly these were too trivial to expose any potential problems that you > may have been referring to, so would be helpful if you could write a more > concrete example / sample code. The trick is to have a sufficiently complicated expression to force the compiler to run out of registers. If the value is non-volatile, it will refetch it (and expect it not to have changed, possibly being disappointed by an interrupt handler running on that same CPU). > > > Or possibly, implement these "volatile" atomic ops variants in inline asm > > > like the patch that Sebastian Siewior has submitted on another thread just > > > a while back. > > > > Given that you are advocating a change (please keep in mind that > > atomic_read() and atomic_set() had volatile semantics on almost all > > platforms), care to give some example where these historical volatile > > semantics are causing a problem? > > [...] > > Can you give even one example > > where the pre-existing volatile semantics are causing enough of a problem > > to justify adding yet more atomic_*() APIs? > > Will take this to the other sub-thread ... OK. > > > Of course, if we find there are more callers in the kernel who want the > > > volatility behaviour than those who don't care, we can re-define the > > > existing ops to such variants, and re-name the existing definitions to > > > somethine else, say "atomic_read_nonvolatile" for all I care. > > > > Do we really need another set of APIs? > > Well, if there's one set of users who do care about volatile behaviour, > and another set that doesn't, it only sounds correct to provide both > those APIs, instead of forcing one behaviour on all users. Well, if the second set doesn't care, they should be OK with the volatile behavior in this case. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 19:07 ` Paul E. McKenney @ 2007-08-15 21:07 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 21:07 UTC (permalink / raw) To: paulmck Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> Possibly these were too trivial to expose any potential problems that >> you >> may have been referring to, so would be helpful if you could write a >> more >> concrete example / sample code. > > The trick is to have a sufficiently complicated expression to force > the compiler to run out of registers. You can use -ffixed-XXX to keep the testcase simple. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 17:55 ` Satyam Sharma 2007-08-15 19:07 ` Paul E. McKenney @ 2007-08-15 20:58 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 20:58 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >>> Of course, if we find there are more callers in the kernel who want >>> the >>> volatility behaviour than those who don't care, we can re-define the >>> existing ops to such variants, and re-name the existing definitions >>> to >>> somethine else, say "atomic_read_nonvolatile" for all I care. >> >> Do we really need another set of APIs? > > Well, if there's one set of users who do care about volatile behaviour, > and another set that doesn't, it only sounds correct to provide both > those APIs, instead of forcing one behaviour on all users. But since there currently is only one such API, and there are users expecting the stronger behaviour, the only sane thing to do is let the API provide that behaviour. You can always add a new API with weaker behaviour later, and move users that are okay with it over to that new API. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 14:25 ` Paul E. McKenney 2007-08-15 15:33 ` Herbert Xu 2007-08-15 17:55 ` Satyam Sharma @ 2007-08-15 18:19 ` David Howells 2007-08-15 18:45 ` Paul E. McKenney 2 siblings, 1 reply; 333+ messages in thread From: David Howells @ 2007-08-15 18:19 UTC (permalink / raw) To: Herbert Xu Cc: dhowells, Paul E. McKenney, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu <herbert@gondor.apana.org.au> wrote: > Let's turn this around. Can you give a single example where > the volatile semantics is needed in a legitimate way? Accessing H/W registers? But apart from that... David ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 18:19 ` David Howells @ 2007-08-15 18:45 ` Paul E. McKenney 2007-08-15 23:41 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 18:45 UTC (permalink / raw) To: David Howells Cc: Herbert Xu, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 07:19:57PM +0100, David Howells wrote: > Herbert Xu <herbert@gondor.apana.org.au> wrote: > > > Let's turn this around. Can you give a single example where > > the volatile semantics is needed in a legitimate way? > > Accessing H/W registers? But apart from that... Communicating between process context and interrupt/NMI handlers using per-CPU variables. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 18:45 ` Paul E. McKenney @ 2007-08-15 23:41 ` Herbert Xu 2007-08-15 23:53 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-15 23:41 UTC (permalink / raw) To: Paul E. McKenney Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 11:45:20AM -0700, Paul E. McKenney wrote: > On Wed, Aug 15, 2007 at 07:19:57PM +0100, David Howells wrote: > > Herbert Xu <herbert@gondor.apana.org.au> wrote: > > > > > Let's turn this around. Can you give a single example where > > > the volatile semantics is needed in a legitimate way? > > > > Accessing H/W registers? But apart from that... > > Communicating between process context and interrupt/NMI handlers using > per-CPU variables. Remeber we're talking about atomic_read/atomic_set. Please cite the actual file/function name you have in mind. Thanks, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 23:41 ` Herbert Xu @ 2007-08-15 23:53 ` Paul E. McKenney 2007-08-16 0:12 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-15 23:53 UTC (permalink / raw) To: Herbert Xu Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 07:41:46AM +0800, Herbert Xu wrote: > On Wed, Aug 15, 2007 at 11:45:20AM -0700, Paul E. McKenney wrote: > > On Wed, Aug 15, 2007 at 07:19:57PM +0100, David Howells wrote: > > > Herbert Xu <herbert@gondor.apana.org.au> wrote: > > > > > > > Let's turn this around. Can you give a single example where > > > > the volatile semantics is needed in a legitimate way? > > > > > > Accessing H/W registers? But apart from that... > > > > Communicating between process context and interrupt/NMI handlers using > > per-CPU variables. > > Remeber we're talking about atomic_read/atomic_set. Please > cite the actual file/function name you have in mind. Yep, we are indeed talking about atomic_read()/atomic_set(). We have been through this issue already in this thread. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 23:53 ` Paul E. McKenney @ 2007-08-16 0:12 ` Herbert Xu 2007-08-16 0:23 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 0:12 UTC (permalink / raw) To: Paul E. McKenney Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote: > > > > Communicating between process context and interrupt/NMI handlers using > > > per-CPU variables. > > > > Remeber we're talking about atomic_read/atomic_set. Please > > cite the actual file/function name you have in mind. > > Yep, we are indeed talking about atomic_read()/atomic_set(). > > We have been through this issue already in this thread. Sorry, but I must've missed it. Could you cite the file or function for my benefit? Thanks, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:12 ` Herbert Xu @ 2007-08-16 0:23 ` Paul E. McKenney 2007-08-16 0:30 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 0:23 UTC (permalink / raw) To: Herbert Xu Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 08:12:48AM +0800, Herbert Xu wrote: > On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote: > > > > > > Communicating between process context and interrupt/NMI handlers using > > > > per-CPU variables. > > > > > > Remeber we're talking about atomic_read/atomic_set. Please > > > cite the actual file/function name you have in mind. > > > > Yep, we are indeed talking about atomic_read()/atomic_set(). > > > > We have been through this issue already in this thread. > > Sorry, but I must've missed it. Could you cite the file or > function for my benefit? I might summarize the thread if there is interest, but I am not able to do so right this minute. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:23 ` Paul E. McKenney @ 2007-08-16 0:30 ` Herbert Xu 2007-08-16 0:49 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 0:30 UTC (permalink / raw) To: Paul E. McKenney Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 05:23:10PM -0700, Paul E. McKenney wrote: > On Thu, Aug 16, 2007 at 08:12:48AM +0800, Herbert Xu wrote: > > On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote: > > > > > > > > Communicating between process context and interrupt/NMI handlers using > > > > > per-CPU variables. > > > > > > > > Remeber we're talking about atomic_read/atomic_set. Please > > > > cite the actual file/function name you have in mind. > > > > > > Yep, we are indeed talking about atomic_read()/atomic_set(). > > > > > > We have been through this issue already in this thread. > > > > Sorry, but I must've missed it. Could you cite the file or > > function for my benefit? > > I might summarize the thread if there is interest, but I am not able to > do so right this minute. Thanks. But I don't need a summary of the thread, I'm asking for an extant code snippet in our kernel that benefits from the volatile change and is not part of a busy-wait. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:30 ` Herbert Xu @ 2007-08-16 0:49 ` Paul E. McKenney 2007-08-16 0:53 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 0:49 UTC (permalink / raw) To: Herbert Xu Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 08:30:23AM +0800, Herbert Xu wrote: > On Wed, Aug 15, 2007 at 05:23:10PM -0700, Paul E. McKenney wrote: > > On Thu, Aug 16, 2007 at 08:12:48AM +0800, Herbert Xu wrote: > > > On Wed, Aug 15, 2007 at 04:53:35PM -0700, Paul E. McKenney wrote: > > > > > > > > > > Communicating between process context and interrupt/NMI handlers using > > > > > > per-CPU variables. > > > > > > > > > > Remeber we're talking about atomic_read/atomic_set. Please > > > > > cite the actual file/function name you have in mind. > > > > > > > > Yep, we are indeed talking about atomic_read()/atomic_set(). > > > > > > > > We have been through this issue already in this thread. > > > > > > Sorry, but I must've missed it. Could you cite the file or > > > function for my benefit? > > > > I might summarize the thread if there is interest, but I am not able to > > do so right this minute. > > Thanks. But I don't need a summary of the thread, I'm asking > for an extant code snippet in our kernel that benefits from > the volatile change and is not part of a busy-wait. Sorry, can't help you there. I really do believe that the information you need (as opposed to the specific item you are asking for) really has been put forth in this thread. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:49 ` Paul E. McKenney @ 2007-08-16 0:53 ` Herbert Xu 2007-08-16 1:14 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 0:53 UTC (permalink / raw) To: Paul E. McKenney Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, Aug 15, 2007 at 05:49:50PM -0700, Paul E. McKenney wrote: > On Thu, Aug 16, 2007 at 08:30:23AM +0800, Herbert Xu wrote: > > > Thanks. But I don't need a summary of the thread, I'm asking > > for an extant code snippet in our kernel that benefits from > > the volatile change and is not part of a busy-wait. > > Sorry, can't help you there. I really do believe that the information > you need (as opposed to the specific item you are asking for) really > has been put forth in this thread. That only leads me to believe that such a code snippet simply does not exist. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:53 ` Herbert Xu @ 2007-08-16 1:14 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 1:14 UTC (permalink / raw) To: Herbert Xu Cc: David Howells, Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 08:53:16AM +0800, Herbert Xu wrote: > On Wed, Aug 15, 2007 at 05:49:50PM -0700, Paul E. McKenney wrote: > > On Thu, Aug 16, 2007 at 08:30:23AM +0800, Herbert Xu wrote: > > > > > Thanks. But I don't need a summary of the thread, I'm asking > > > for an extant code snippet in our kernel that benefits from > > > the volatile change and is not part of a busy-wait. > > > > Sorry, can't help you there. I really do believe that the information > > you need (as opposed to the specific item you are asking for) really > > has been put forth in this thread. > > That only leads me to believe that such a code snippet simply > does not exist. Whatever... Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 12:31 ` Satyam Sharma 2007-08-15 13:08 ` Stefan Richter @ 2007-08-15 18:31 ` Segher Boessenkool 2007-08-15 19:40 ` Satyam Sharma 2007-08-15 23:22 ` Paul Mackerras 2007-08-16 3:37 ` Bill Fink 3 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 18:31 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang > "Volatile behaviour" itself isn't consistently defined (at least > definitely not consistently implemented in various gcc versions across > platforms), It should be consistent across platforms; if not, file a bug please. > but it is /expected/ to mean something like: "ensure that > every such access actually goes all the way to memory, and is not > re-ordered w.r.t. to other accesses, as far as the compiler can take > care of these". The last "as far as compiler can take care" disclaimer > comes about due to CPUs doing their own re-ordering nowadays. You can *expect* whatever you want, but this isn't in line with reality at all. volatile _does not_ make accesses go all the way to memory. volatile _does not_ prevent reordering wrt other accesses. What volatile does are a) never optimise away a read (or write) to the object, since the data can change in ways the compiler cannot see; and b) never move stores to the object across a sequence point. This does not mean other accesses cannot be reordered wrt the volatile access. If the abstract machine would do an access to a volatile- qualified object, the generated machine code will do that access too. But, for example, it can still be optimised away by the compiler, if it can prove it is allowed to. If you want stuff to go all the way to memory, you need some architecture-specific flush sequence; to make a store globally visible before another store, you need mb(); before some following read, you need mb(); to prevent reordering you need a barrier. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 18:31 ` Segher Boessenkool @ 2007-08-15 19:40 ` Satyam Sharma 2007-08-15 20:42 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-15 19:40 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Wed, 15 Aug 2007, Segher Boessenkool wrote: > > "Volatile behaviour" itself isn't consistently defined (at least > > definitely not consistently implemented in various gcc versions across > > platforms), > > It should be consistent across platforms; if not, file a bug please. > > > but it is /expected/ to mean something like: "ensure that > > every such access actually goes all the way to memory, and is not > > re-ordered w.r.t. to other accesses, as far as the compiler can take ^ (volatile) (or, alternatively, "other accesses to the same volatile object" ...) > > care of these". The last "as far as compiler can take care" disclaimer > > comes about due to CPUs doing their own re-ordering nowadays. > > You can *expect* whatever you want, but this isn't in line with > reality at all. > > volatile _does not_ prevent reordering wrt other accesses. > [...] > What volatile does are a) never optimise away a read (or write) > to the object, since the data can change in ways the compiler > cannot see; and b) never move stores to the object across a > sequence point. This does not mean other accesses cannot be > reordered wrt the volatile access. > > If the abstract machine would do an access to a volatile- > qualified object, the generated machine code will do that > access too. But, for example, it can still be optimised > away by the compiler, if it can prove it is allowed to. As (now) indicated above, I had meant multiple volatile accesses to the same object, obviously. BTW: #define atomic_read(a) (*(volatile int *)&(a)) #define atomic_set(a,i) (*(volatile int *)&(a) = (i)) int a; void func(void) { int b; b = atomic_read(a); atomic_set(a, 20); b = atomic_read(a); } gives: func: pushl %ebp movl a, %eax movl %esp, %ebp movl $20, a movl a, %eax popl %ebp ret so the first atomic_read() wasn't optimized away. > volatile _does not_ make accesses go all the way to memory. > [...] > If you want stuff to go all the way to memory, you need some > architecture-specific flush sequence; to make a store globally > visible before another store, you need mb(); before some following > read, you need mb(); to prevent reordering you need a barrier. Sure, which explains the "as far as the compiler can take care" bit. Poor phrase / choice of words, probably. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 19:40 ` Satyam Sharma @ 2007-08-15 20:42 ` Segher Boessenkool 2007-08-16 1:23 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-15 20:42 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> What volatile does are a) never optimise away a read (or write) >> to the object, since the data can change in ways the compiler >> cannot see; and b) never move stores to the object across a >> sequence point. This does not mean other accesses cannot be >> reordered wrt the volatile access. >> >> If the abstract machine would do an access to a volatile- >> qualified object, the generated machine code will do that >> access too. But, for example, it can still be optimised >> away by the compiler, if it can prove it is allowed to. > > As (now) indicated above, I had meant multiple volatile accesses to > the same object, obviously. Yes, accesses to volatile objects are never reordered with respect to each other. > BTW: > > #define atomic_read(a) (*(volatile int *)&(a)) > #define atomic_set(a,i) (*(volatile int *)&(a) = (i)) > > int a; > > void func(void) > { > int b; > > b = atomic_read(a); > atomic_set(a, 20); > b = atomic_read(a); > } > > gives: > > func: > pushl %ebp > movl a, %eax > movl %esp, %ebp > movl $20, a > movl a, %eax > popl %ebp > ret > > so the first atomic_read() wasn't optimized away. Of course. It is executed by the abstract machine, so it will be executed by the actual machine. On the other hand, try b = 0; if (b) b = atomic_read(a); or similar. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 20:42 ` Segher Boessenkool @ 2007-08-16 1:23 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 1:23 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Wed, 15 Aug 2007, Segher Boessenkool wrote: > [...] > > BTW: > > > > #define atomic_read(a) (*(volatile int *)&(a)) > > #define atomic_set(a,i) (*(volatile int *)&(a) = (i)) > > > > int a; > > > > void func(void) > > { > > int b; > > > > b = atomic_read(a); > > atomic_set(a, 20); > > b = atomic_read(a); > > } > > > > gives: > > > > func: > > pushl %ebp > > movl a, %eax > > movl %esp, %ebp > > movl $20, a > > movl a, %eax > > popl %ebp > > ret > > > > so the first atomic_read() wasn't optimized away. > > Of course. It is executed by the abstract machine, so > it will be executed by the actual machine. On the other > hand, try > > b = 0; > if (b) > b = atomic_read(a); > > or similar. Yup, obviously. Volatile accesses (or any access to volatile objects), or even "__volatile__ asms" (which gcc normally promises never to elid) can always be optimized for cases such as these where the compiler can trivially determine that the code in question is not reachable. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 12:31 ` Satyam Sharma 2007-08-15 13:08 ` Stefan Richter 2007-08-15 18:31 ` Segher Boessenkool @ 2007-08-15 23:22 ` Paul Mackerras 2007-08-16 0:26 ` Christoph Lameter 2007-08-24 12:50 ` Denys Vlasenko 2007-08-16 3:37 ` Bill Fink 3 siblings, 2 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-15 23:22 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney Satyam Sharma writes: > > Doesn't "atomic WRT all processors" require volatility? > > No, it definitely doesn't. Why should it? > > "Atomic w.r.t. all processors" is just your normal, simple "atomicity" > for SMP systems (ensure that that object is modified / set / replaced > in main memory atomically) and has nothing to do with "volatile" > behaviour. Atomic variables are "volatile" in the sense that they are liable to be changed at any time by mechanisms that are outside the knowledge of the C compiler, namely, other CPUs, or this CPU executing an interrupt routine. In the kernel we use atomic variables in precisely those situations where a variable is potentially accessed concurrently by multiple CPUs, and where each CPU needs to see updates done by other CPUs in a timely fashion. That is what they are for. Therefore the compiler must not cache values of atomic variables in registers; each atomic_read must result in a load and each atomic_set must result in a store. Anything else will just lead to subtle bugs. I have no strong opinion about whether or not the best way to achieve this is through the use of the "volatile" C keyword. Segher's idea of using asm instead seems like a good one to me. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 23:22 ` Paul Mackerras @ 2007-08-16 0:26 ` Christoph Lameter 2007-08-16 0:34 ` Paul Mackerras 2007-08-16 0:39 ` Paul E. McKenney 2007-08-24 12:50 ` Denys Vlasenko 1 sibling, 2 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 0:26 UTC (permalink / raw) To: Paul Mackerras Cc: Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Thu, 16 Aug 2007, Paul Mackerras wrote: > In the kernel we use atomic variables in precisely those situations > where a variable is potentially accessed concurrently by multiple > CPUs, and where each CPU needs to see updates done by other CPUs in a > timely fashion. That is what they are for. Therefore the compiler > must not cache values of atomic variables in registers; each > atomic_read must result in a load and each atomic_set must result in a > store. Anything else will just lead to subtle bugs. This may have been the intend. However, today the visibility is controlled using barriers. And we have barriers that we use with atomic operations. Having volatile be the default just lead to confusion. Atomic read should just read with no extras. Extras can be added by using variants like atomic_read_volatile or so. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:26 ` Christoph Lameter @ 2007-08-16 0:34 ` Paul Mackerras 2007-08-16 0:40 ` Christoph Lameter 2007-08-16 0:39 ` Paul E. McKenney 1 sibling, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 0:34 UTC (permalink / raw) To: Christoph Lameter Cc: Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney Christoph Lameter writes: > On Thu, 16 Aug 2007, Paul Mackerras wrote: > > > In the kernel we use atomic variables in precisely those situations > > where a variable is potentially accessed concurrently by multiple > > CPUs, and where each CPU needs to see updates done by other CPUs in a > > timely fashion. That is what they are for. Therefore the compiler > > must not cache values of atomic variables in registers; each > > atomic_read must result in a load and each atomic_set must result in a > > store. Anything else will just lead to subtle bugs. > > This may have been the intend. However, today the visibility is controlled > using barriers. And we have barriers that we use with atomic operations. Those barriers are for when we need ordering between atomic variables and other memory locations. An atomic variable by itself doesn't and shouldn't need any barriers for other CPUs to be able to see what's happening to it. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:34 ` Paul Mackerras @ 2007-08-16 0:40 ` Christoph Lameter 0 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 0:40 UTC (permalink / raw) To: Paul Mackerras Cc: Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Thu, 16 Aug 2007, Paul Mackerras wrote: > Those barriers are for when we need ordering between atomic variables > and other memory locations. An atomic variable by itself doesn't and > shouldn't need any barriers for other CPUs to be able to see what's > happening to it. It does not need any barriers. As soon as one cpu acquires the cacheline for write it will be invalidated in the caches of the others. So the other cpu will have to refetch. No need for volatile. The issue here may be that the compiler has fetched the atomic variable earlier and put it into a register. However, that prefetching is limited because it cannot cross functions calls etc. The only problem could be loops where the compiler does not refetch the variable since it assumes that it does not change and there are no function calls in the body of the loop. But AFAIK these loops need cpu_relax and other measures anyways to avoid bad effects from busy waiting. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:26 ` Christoph Lameter 2007-08-16 0:34 ` Paul Mackerras @ 2007-08-16 0:39 ` Paul E. McKenney 2007-08-16 0:42 ` Christoph Lameter 1 sibling, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 0:39 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, Aug 15, 2007 at 05:26:34PM -0700, Christoph Lameter wrote: > On Thu, 16 Aug 2007, Paul Mackerras wrote: > > > In the kernel we use atomic variables in precisely those situations > > where a variable is potentially accessed concurrently by multiple > > CPUs, and where each CPU needs to see updates done by other CPUs in a > > timely fashion. That is what they are for. Therefore the compiler > > must not cache values of atomic variables in registers; each > > atomic_read must result in a load and each atomic_set must result in a > > store. Anything else will just lead to subtle bugs. > > This may have been the intend. However, today the visibility is controlled > using barriers. And we have barriers that we use with atomic operations. > Having volatile be the default just lead to confusion. Atomic read should > just read with no extras. Extras can be added by using variants like > atomic_read_volatile or so. Seems to me that we face greater chance of confusion without the volatile than with, particularly as compiler optimizations become more aggressive. Yes, we could simply disable optimization, but optimization can be quite helpful. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:39 ` Paul E. McKenney @ 2007-08-16 0:42 ` Christoph Lameter 2007-08-16 0:53 ` Paul E. McKenney 2007-08-16 1:51 ` Paul Mackerras 0 siblings, 2 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 0:42 UTC (permalink / raw) To: Paul E. McKenney Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, 15 Aug 2007, Paul E. McKenney wrote: > Seems to me that we face greater chance of confusion without the > volatile than with, particularly as compiler optimizations become > more aggressive. Yes, we could simply disable optimization, but > optimization can be quite helpful. A volatile default would disable optimizations for atomic_read. atomic_read without volatile would allow for full optimization by the compiler. Seems that this is what one wants in many cases. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:42 ` Christoph Lameter @ 2007-08-16 0:53 ` Paul E. McKenney 2007-08-16 0:59 ` Christoph Lameter 2007-08-16 1:51 ` Paul Mackerras 1 sibling, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 0:53 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, Aug 15, 2007 at 05:42:07PM -0700, Christoph Lameter wrote: > On Wed, 15 Aug 2007, Paul E. McKenney wrote: > > > Seems to me that we face greater chance of confusion without the > > volatile than with, particularly as compiler optimizations become > > more aggressive. Yes, we could simply disable optimization, but > > optimization can be quite helpful. > > A volatile default would disable optimizations for atomic_read. > atomic_read without volatile would allow for full optimization by the > compiler. Seems that this is what one wants in many cases. The volatile cast should not disable all that many optimizations, for example, it is much less hurtful than barrier(). Furthermore, the main optimizations disabled (pulling atomic_read() and atomic_set() out of loops) really do need to be disabled. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:53 ` Paul E. McKenney @ 2007-08-16 0:59 ` Christoph Lameter 2007-08-16 1:14 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 0:59 UTC (permalink / raw) To: Paul E. McKenney Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, 15 Aug 2007, Paul E. McKenney wrote: > The volatile cast should not disable all that many optimizations, > for example, it is much less hurtful than barrier(). Furthermore, > the main optimizations disabled (pulling atomic_read() and atomic_set() > out of loops) really do need to be disabled. In many cases you do not need a barrier. Having volatile there *will* impact optimization because the compiler cannot use a register that may contain the value that was fetched earlier. And the compiler cannot choose freely when to fetch the value. The order of memory accesses are fixed if you use volatile. If the variable is not volatile then the compiler can arrange memory accesses any way they fit and thus generate better code. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:59 ` Christoph Lameter @ 2007-08-16 1:14 ` Paul E. McKenney 2007-08-16 1:41 ` Christoph Lameter 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 1:14 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, Aug 15, 2007 at 05:59:41PM -0700, Christoph Lameter wrote: > On Wed, 15 Aug 2007, Paul E. McKenney wrote: > > > The volatile cast should not disable all that many optimizations, > > for example, it is much less hurtful than barrier(). Furthermore, > > the main optimizations disabled (pulling atomic_read() and atomic_set() > > out of loops) really do need to be disabled. > > In many cases you do not need a barrier. Having volatile there *will* > impact optimization because the compiler cannot use a register that may > contain the value that was fetched earlier. And the compiler cannot choose > freely when to fetch the value. The order of memory accesses are fixed if > you use volatile. If the variable is not volatile then the compiler can > arrange memory accesses any way they fit and thus generate better code. Understood. My point is not that the impact is precisely zero, but rather that the impact on optimization is much less hurtful than the problems that could arise otherwise, particularly as compilers become more aggressive in their optimizations. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:14 ` Paul E. McKenney @ 2007-08-16 1:41 ` Christoph Lameter 2007-08-16 2:15 ` Satyam Sharma 2007-08-16 2:32 ` Paul E. McKenney 0 siblings, 2 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 1:41 UTC (permalink / raw) To: Paul E. McKenney Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, 15 Aug 2007, Paul E. McKenney wrote: > Understood. My point is not that the impact is precisely zero, but > rather that the impact on optimization is much less hurtful than the > problems that could arise otherwise, particularly as compilers become > more aggressive in their optimizations. The problems arise because barriers are not used as required. Volatile has wishy washy semantics and somehow marries memory barriers with data access. It is clearer to separate the two. Conceptual cleanness usually translates into better code. If one really wants the volatile then lets make it explicit and use atomic_read_volatile() ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:41 ` Christoph Lameter @ 2007-08-16 2:15 ` Satyam Sharma 2007-08-16 2:08 ` Herbert Xu 2007-08-16 2:32 ` Paul E. McKenney 1 sibling, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 2:15 UTC (permalink / raw) To: Christoph Lameter Cc: Paul E. McKenney, Paul Mackerras, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, 15 Aug 2007, Christoph Lameter wrote: > On Wed, 15 Aug 2007, Paul E. McKenney wrote: > > > Understood. My point is not that the impact is precisely zero, but > > rather that the impact on optimization is much less hurtful than the > > problems that could arise otherwise, particularly as compilers become > > more aggressive in their optimizations. > > The problems arise because barriers are not used as required. Volatile > has wishy washy semantics and somehow marries memory barriers with data > access. It is clearer to separate the two. Conceptual cleanness usually > translates into better code. If one really wants the volatile then lets > make it explicit and use > > atomic_read_volatile() Completely agreed, again. To summarize again (had done so about ~100 mails earlier in this thread too :-) ... atomic_{read,set}_volatile() -- guarantees volatility also along with atomicity (the two _are_ different concepts after all, irrespective of whether callsites normally want one with the other or not) atomic_{read,set}_nonvolatile() -- only guarantees atomicity, compiler free to elid / coalesce / optimize such accesses, can keep the object in question cached in a local register, leads to smaller text, etc. As to which one should be the default atomic_read() is a question of whether majority of callsites (more weightage to important / hot codepaths, lesser to obscure callsites) want a particular behaviour. Do we have a consensus here? (hoping against hope, probably :-) [ This thread has gotten completely out of hand ... for my mail client alpine as well, it now seems. Reminds of that 1000+ GPLv3 fest :-) ] ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:15 ` Satyam Sharma @ 2007-08-16 2:08 ` Herbert Xu 2007-08-16 2:18 ` Christoph Lameter 2007-08-16 2:18 ` Chris Friesen 0 siblings, 2 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-16 2:08 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul E. McKenney, Paul Mackerras, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 07:45:44AM +0530, Satyam Sharma wrote: > > Completely agreed, again. To summarize again (had done so about ~100 mails > earlier in this thread too :-) ... > > atomic_{read,set}_volatile() -- guarantees volatility also along with > atomicity (the two _are_ different concepts after all, irrespective of > whether callsites normally want one with the other or not) > > atomic_{read,set}_nonvolatile() -- only guarantees atomicity, compiler > free to elid / coalesce / optimize such accesses, can keep the object > in question cached in a local register, leads to smaller text, etc. > > As to which one should be the default atomic_read() is a question of > whether majority of callsites (more weightage to important / hot > codepaths, lesser to obscure callsites) want a particular behaviour. > > Do we have a consensus here? (hoping against hope, probably :-) I can certainly agree with this. But I have to say that I still don't know of a single place where one would actually use the volatile variant. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:08 ` Herbert Xu @ 2007-08-16 2:18 ` Christoph Lameter 2007-08-16 3:23 ` Paul Mackerras 2007-08-16 2:18 ` Chris Friesen 1 sibling, 1 reply; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 2:18 UTC (permalink / raw) To: Herbert Xu Cc: Satyam Sharma, Paul E. McKenney, Paul Mackerras, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Herbert Xu wrote: > > Do we have a consensus here? (hoping against hope, probably :-) > > I can certainly agree with this. I agree too. > But I have to say that I still don't know of a single place > where one would actually use the volatile variant. I suspect that what you say is true after we have looked at all callers. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:18 ` Christoph Lameter @ 2007-08-16 3:23 ` Paul Mackerras 2007-08-16 3:33 ` Herbert Xu ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 3:23 UTC (permalink / raw) To: Christoph Lameter Cc: Herbert Xu, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Christoph Lameter writes: > > But I have to say that I still don't know of a single place > > where one would actually use the volatile variant. > > I suspect that what you say is true after we have looked at all callers. It seems that there could be a lot of places where atomic_t is used in a non-atomic fashion, and that those uses are either buggy, or there is some lock held at the time which guarantees that other CPUs aren't changing the value. In both cases there is no point in using atomic_t; we might as well just use an ordinary int. In particular, atomic_read seems to lend itself to buggy uses. People seem to do things like: atomic_add(&v, something); if (atomic_read(&v) > something_else) ... and expect that there is some relationship between the value that the atomic_add stored and the value that the atomic_read will return, which there isn't. People seem to think that using atomic_t magically gets rid of races. It doesn't. I'd go so far as to say that anywhere where you want a non-"volatile" atomic_read, either your code is buggy, or else an int would work just as well. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:23 ` Paul Mackerras @ 2007-08-16 3:33 ` Herbert Xu 2007-08-16 3:48 ` Paul Mackerras 2007-08-16 18:48 ` Christoph Lameter 2007-08-16 19:44 ` Segher Boessenkool 2 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 3:33 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 01:23:06PM +1000, Paul Mackerras wrote: > > In particular, atomic_read seems to lend itself to buggy uses. People > seem to do things like: > > atomic_add(&v, something); > if (atomic_read(&v) > something_else) ... If you're referring to the code in sk_stream_mem_schedule then it's working as intended. The atomicity guarantees that the atomic_add/atomic_sub won't be seen in parts by other readers. We certainly do not need to see other atomic_add/atomic_sub operations immediately. If you're referring to another code snippet please cite. > I'd go so far as to say that anywhere where you want a non-"volatile" > atomic_read, either your code is buggy, or else an int would work just > as well. An int won't work here because += and -= do not have the atomicity guarantees that atomic_add/atomic_sub do. In particular, this may cause an atomic_read on another CPU to give a bogus reading. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:33 ` Herbert Xu @ 2007-08-16 3:48 ` Paul Mackerras 2007-08-16 4:03 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 3:48 UTC (permalink / raw) To: Herbert Xu Cc: Christoph Lameter, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > If you're referring to the code in sk_stream_mem_schedule > then it's working as intended. The atomicity guarantees You mean it's intended that *sk->sk_prot->memory_pressure can end up as 1 when sk->sk_prot->memory_allocated is small (less than ->sysctl_mem[0]), or as 0 when ->memory_allocated is large (greater than ->sysctl_mem[2])? Because that's the effect of the current code. If so I wonder why you bother computing it. > that the atomic_add/atomic_sub won't be seen in parts by > other readers. > > We certainly do not need to see other atomic_add/atomic_sub > operations immediately. > > If you're referring to another code snippet please cite. > > > I'd go so far as to say that anywhere where you want a non-"volatile" > > atomic_read, either your code is buggy, or else an int would work just > > as well. > > An int won't work here because += and -= do not have the > atomicity guarantees that atomic_add/atomic_sub do. In > particular, this may cause an atomic_read on another CPU > to give a bogus reading. The point is that guaranteeing the atomicity of the increment or decrement does not suffice to make the code race-free. In this case the race arises from the fact that reading ->memory_allocated and setting *->memory_pressure are separate operations. To make that code work properly you need a lock. And once you have the lock an ordinary int would suffice for ->memory_allocated. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:48 ` Paul Mackerras @ 2007-08-16 4:03 ` Herbert Xu 2007-08-16 4:34 ` Paul Mackerras 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 4:03 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 01:48:32PM +1000, Paul Mackerras wrote: > Herbert Xu writes: > > > If you're referring to the code in sk_stream_mem_schedule > > then it's working as intended. The atomicity guarantees > > You mean it's intended that *sk->sk_prot->memory_pressure can end up > as 1 when sk->sk_prot->memory_allocated is small (less than > ->sysctl_mem[0]), or as 0 when ->memory_allocated is large (greater > than ->sysctl_mem[2])? Because that's the effect of the current code. > If so I wonder why you bother computing it. You need to remember that there are three different limits: minimum, pressure, and maximum. By default we should never be in a situation where what you say can occur. If you set all three limits to the same thing, then yes it won't work as intended but it's still well-behaved. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 4:03 ` Herbert Xu @ 2007-08-16 4:34 ` Paul Mackerras 2007-08-16 5:37 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 4:34 UTC (permalink / raw) To: Herbert Xu Cc: Christoph Lameter, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > > You mean it's intended that *sk->sk_prot->memory_pressure can end up > > as 1 when sk->sk_prot->memory_allocated is small (less than > > ->sysctl_mem[0]), or as 0 when ->memory_allocated is large (greater > > than ->sysctl_mem[2])? Because that's the effect of the current code. > > If so I wonder why you bother computing it. > > You need to remember that there are three different limits: > minimum, pressure, and maximum. By default we should never > be in a situation where what you say can occur. > > If you set all three limits to the same thing, then yes it > won't work as intended but it's still well-behaved. I'm not talking about setting all three limits to the same thing. I'm talking about this situation: CPU 0 comes into __sk_stream_mem_reclaim, reads memory_allocated, but then before it can do the store to *memory_pressure, CPUs 1-1023 all go through sk_stream_mem_schedule, collectively increase memory_allocated to more than sysctl_mem[2] and set *memory_pressure. Finally CPU 0 gets to do its store and it sets *memory_pressure back to 0, but by this stage memory_allocated is way larger than sysctl_mem[2]. Yes, it's unlikely, but that is the nature of race conditions - they are unlikely, and only show up at inconvenient times, never when someone who could fix the bug is watching. :) Similarly it would be possible for other CPUs to decrease memory_allocated from greater than sysctl_mem[2] to less than sysctl_mem[0] in the interval between when we read memory_allocated and set *memory_pressure to 1. And it's quite possible for their setting of *memory_pressure to 0 to happen before our setting of it to 1, so that it ends up at 1 when it should be 0. Now, maybe it's the case that it doesn't really matter whether *->memory_pressure is 0 or 1. But if so, why bother computing it at all? People seem to think that using atomic_t means they don't need to use a spinlock. That's fine if there is only one variable involved, but as soon as there's more than one, there's the possibility of a race, whether or not you use atomic_t, and whether or not atomic_read has "volatile" behaviour. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 4:34 ` Paul Mackerras @ 2007-08-16 5:37 ` Herbert Xu 2007-08-16 6:00 ` Paul Mackerras 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 5:37 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 02:34:25PM +1000, Paul Mackerras wrote: > > I'm talking about this situation: > > CPU 0 comes into __sk_stream_mem_reclaim, reads memory_allocated, but > then before it can do the store to *memory_pressure, CPUs 1-1023 all > go through sk_stream_mem_schedule, collectively increase > memory_allocated to more than sysctl_mem[2] and set *memory_pressure. > Finally CPU 0 gets to do its store and it sets *memory_pressure back > to 0, but by this stage memory_allocated is way larger than > sysctl_mem[2]. It doesn't matter. The memory pressure flag is an *advisory* flag. If we get it wrong the worst that'll happen is that we'd waste some time doing work that'll be thrown away. Please look at the places where it's used before jumping to conclusions. > Now, maybe it's the case that it doesn't really matter whether > *->memory_pressure is 0 or 1. But if so, why bother computing it at > all? As long as we get it right most of the time (and I think you would agree that we do get it right most of the time), then this flag has achieved its purpose. > People seem to think that using atomic_t means they don't need to use > a spinlock. That's fine if there is only one variable involved, but > as soon as there's more than one, there's the possibility of a race, > whether or not you use atomic_t, and whether or not atomic_read has > "volatile" behaviour. In any case, this actually illustrates why the addition of volatile is completely pointless. Even if this code was broken, which it definitely is not, having the volatile there wouldn't have helped at all. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 5:37 ` Herbert Xu @ 2007-08-16 6:00 ` Paul Mackerras 2007-08-16 18:50 ` Christoph Lameter 0 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 6:00 UTC (permalink / raw) To: Herbert Xu Cc: Christoph Lameter, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > It doesn't matter. The memory pressure flag is an *advisory* > flag. If we get it wrong the worst that'll happen is that we'd > waste some time doing work that'll be thrown away. Ah, so it's the "racy but I don't care because it's only an optimization" case. That's fine. Somehow I find it hard to believe that all the racy uses of atomic_read in the kernel are like that, though. :) > In any case, this actually illustrates why the addition of > volatile is completely pointless. Even if this code was > broken, which it definitely is not, having the volatile > there wouldn't have helped at all. Yes, adding volatile to racy code doesn't somehow make it race-free. Neither does using atomic_t, despite what some seem to believe. I have actually started going through all the uses of atomic_read in the kernel. So far out of the first 100 I have found none where we have two atomic_reads of the same variable and the compiler could usefully use the value from the first as the result of the second. But there's still > 2500 to go... Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 6:00 ` Paul Mackerras @ 2007-08-16 18:50 ` Christoph Lameter 0 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 18:50 UTC (permalink / raw) To: Paul Mackerras Cc: Herbert Xu, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Paul Mackerras wrote: > Herbert Xu writes: > > > It doesn't matter. The memory pressure flag is an *advisory* > > flag. If we get it wrong the worst that'll happen is that we'd > > waste some time doing work that'll be thrown away. > > Ah, so it's the "racy but I don't care because it's only an > optimization" case. That's fine. Somehow I find it hard to believe > that all the racy uses of atomic_read in the kernel are like that, > though. :) My use of atomic_read in SLUB is like that. Volatile does not magically sync up reads somehow. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:23 ` Paul Mackerras 2007-08-16 3:33 ` Herbert Xu @ 2007-08-16 18:48 ` Christoph Lameter 2007-08-16 19:44 ` Segher Boessenkool 2 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 18:48 UTC (permalink / raw) To: Paul Mackerras Cc: Herbert Xu, Satyam Sharma, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Paul Mackerras wrote: > > It seems that there could be a lot of places where atomic_t is used in > a non-atomic fashion, and that those uses are either buggy, or there > is some lock held at the time which guarantees that other CPUs aren't > changing the value. In both cases there is no point in using > atomic_t; we might as well just use an ordinary int. The point of atomic_t is to do atomic *changes* to the variable. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:23 ` Paul Mackerras 2007-08-16 3:33 ` Herbert Xu 2007-08-16 18:48 ` Christoph Lameter @ 2007-08-16 19:44 ` Segher Boessenkool 2 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 19:44 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang > I'd go so far as to say that anywhere where you want a non-"volatile" > atomic_read, either your code is buggy, or else an int would work just > as well. Even, the only way to implement a "non-volatile" atomic_read() is essentially as a plain int (you can do some tricks so you cannot assign to the result and stuff like that, but that's not the issue here). So if that would be the behaviour we wanted, just get rid of that whole atomic_read() thing, so no one can misuse it anymore. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:08 ` Herbert Xu 2007-08-16 2:18 ` Christoph Lameter @ 2007-08-16 2:18 ` Chris Friesen 1 sibling, 0 replies; 333+ messages in thread From: Chris Friesen @ 2007-08-16 2:18 UTC (permalink / raw) To: Herbert Xu Cc: Satyam Sharma, Christoph Lameter, Paul E. McKenney, Paul Mackerras, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > But I have to say that I still don't know of a single place > where one would actually use the volatile variant. Given that many of the existing users do currently have "volatile", are you comfortable simply removing that behaviour from them? Are you sure that you will not introduce any issues? Forcing a re-read is only a performance penalty. Removing it can cause behavioural changes. I would be more comfortable making the default match the majority of the current implementations (ie: volatile semantics). Then, if someone cares about performance they can explicitly validate the call path and convert it over to the non-volatile version. Correctness before speed... Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:41 ` Christoph Lameter 2007-08-16 2:15 ` Satyam Sharma @ 2007-08-16 2:32 ` Paul E. McKenney 1 sibling, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 2:32 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu On Wed, Aug 15, 2007 at 06:41:40PM -0700, Christoph Lameter wrote: > On Wed, 15 Aug 2007, Paul E. McKenney wrote: > > > Understood. My point is not that the impact is precisely zero, but > > rather that the impact on optimization is much less hurtful than the > > problems that could arise otherwise, particularly as compilers become > > more aggressive in their optimizations. > > The problems arise because barriers are not used as required. Volatile > has wishy washy semantics and somehow marries memory barriers with data > access. It is clearer to separate the two. Conceptual cleanness usually > translates into better code. If one really wants the volatile then lets > make it explicit and use > > atomic_read_volatile() There are indeed architectures where you can cause gcc to emit memory barriers in response to volatile. I am assuming that we are -not- making gcc do this. Given this, then volatiles and memory barrier instructions are orthogonal -- one controls the compiler, the other controls the CPU. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 0:42 ` Christoph Lameter 2007-08-16 0:53 ` Paul E. McKenney @ 2007-08-16 1:51 ` Paul Mackerras 2007-08-16 2:00 ` Herbert Xu 2007-08-16 2:07 ` Segher Boessenkool 1 sibling, 2 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 1:51 UTC (permalink / raw) To: Christoph Lameter Cc: Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu Christoph Lameter writes: > A volatile default would disable optimizations for atomic_read. > atomic_read without volatile would allow for full optimization by the > compiler. Seems that this is what one wants in many cases. Name one such case. An atomic_read should do a load from memory. If the programmer puts an atomic_read() in the code then the compiler should emit a load for it, not re-use a value returned by a previous atomic_read. I do not believe it would ever be useful for the compiler to collapse two atomic_read statements into a single load. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:51 ` Paul Mackerras @ 2007-08-16 2:00 ` Herbert Xu 2007-08-16 2:05 ` Paul Mackerras 2007-08-16 2:07 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 2:00 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 11:51:42AM +1000, Paul Mackerras wrote: > > Name one such case. See sk_stream_mem_schedule in net/core/stream.c: /* Under limit. */ if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) { if (*sk->sk_prot->memory_pressure) *sk->sk_prot->memory_pressure = 0; return 1; } /* Over hard limit. */ if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) { sk->sk_prot->enter_memory_pressure(); goto suppress_allocation; } We don't need to reload sk->sk_prot->memory_allocated here. Now where is your example again? Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:00 ` Herbert Xu @ 2007-08-16 2:05 ` Paul Mackerras 2007-08-16 2:11 ` Herbert Xu ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 2:05 UTC (permalink / raw) To: Herbert Xu Cc: Christoph Lameter, Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > See sk_stream_mem_schedule in net/core/stream.c: > > /* Under limit. */ > if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) { > if (*sk->sk_prot->memory_pressure) > *sk->sk_prot->memory_pressure = 0; > return 1; > } > > /* Over hard limit. */ > if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) { > sk->sk_prot->enter_memory_pressure(); > goto suppress_allocation; > } > > We don't need to reload sk->sk_prot->memory_allocated here. Are you sure? How do you know some other CPU hasn't changed the value in between? Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:05 ` Paul Mackerras @ 2007-08-16 2:11 ` Herbert Xu 2007-08-16 2:35 ` Paul E. McKenney 2007-08-16 3:15 ` Paul Mackerras 2007-08-16 2:15 ` Christoph Lameter 2007-08-16 2:33 ` Satyam Sharma 2 siblings, 2 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-16 2:11 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 12:05:56PM +1000, Paul Mackerras wrote: > Herbert Xu writes: > > > See sk_stream_mem_schedule in net/core/stream.c: > > > > /* Under limit. */ > > if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) { > > if (*sk->sk_prot->memory_pressure) > > *sk->sk_prot->memory_pressure = 0; > > return 1; > > } > > > > /* Over hard limit. */ > > if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) { > > sk->sk_prot->enter_memory_pressure(); > > goto suppress_allocation; > > } > > > > We don't need to reload sk->sk_prot->memory_allocated here. > > Are you sure? How do you know some other CPU hasn't changed the value > in between? Yes I'm sure, because we don't care if others have increased the reservation. Note that even if we did we'd be using barriers so volatile won't do us any good here. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:11 ` Herbert Xu @ 2007-08-16 2:35 ` Paul E. McKenney 2007-08-16 3:15 ` Paul Mackerras 1 sibling, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 2:35 UTC (permalink / raw) To: Herbert Xu Cc: Paul Mackerras, Christoph Lameter, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 10:11:05AM +0800, Herbert Xu wrote: > On Thu, Aug 16, 2007 at 12:05:56PM +1000, Paul Mackerras wrote: > > Herbert Xu writes: > > > > > See sk_stream_mem_schedule in net/core/stream.c: > > > > > > /* Under limit. */ > > > if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) { > > > if (*sk->sk_prot->memory_pressure) > > > *sk->sk_prot->memory_pressure = 0; > > > return 1; > > > } > > > > > > /* Over hard limit. */ > > > if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) { > > > sk->sk_prot->enter_memory_pressure(); > > > goto suppress_allocation; > > > } > > > > > > We don't need to reload sk->sk_prot->memory_allocated here. > > > > Are you sure? How do you know some other CPU hasn't changed the value > > in between? > > Yes I'm sure, because we don't care if others have increased > the reservation. > > Note that even if we did we'd be using barriers so volatile > won't do us any good here. If the load-coalescing is important to performance, why not load into a local variable? Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:11 ` Herbert Xu 2007-08-16 2:35 ` Paul E. McKenney @ 2007-08-16 3:15 ` Paul Mackerras 2007-08-16 3:43 ` Herbert Xu 1 sibling, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 3:15 UTC (permalink / raw) To: Herbert Xu Cc: Christoph Lameter, Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > > Are you sure? How do you know some other CPU hasn't changed the value > > in between? > > Yes I'm sure, because we don't care if others have increased > the reservation. But others can also reduce the reservation. Also, the code sets and clears *sk->sk_prot->memory_pressure nonatomically with respect to the reads of sk->sk_prot->memory_allocated, so in fact the code doesn't guarantee any particular relationship between the two. That code looks like a beautiful example of buggy, racy code where someone has sprinkled magic fix-the-races dust (otherwise known as atomic_t) around in a vain attempt to fix the races. That's assuming that all that stuff actually performs any useful purpose, of course, and that there isn't some lock held by the callers. In the latter case it is pointless using atomic_t. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:15 ` Paul Mackerras @ 2007-08-16 3:43 ` Herbert Xu 0 siblings, 0 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-16 3:43 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 01:15:05PM +1000, Paul Mackerras wrote: > > But others can also reduce the reservation. Also, the code sets and > clears *sk->sk_prot->memory_pressure nonatomically with respect to the > reads of sk->sk_prot->memory_allocated, so in fact the code doesn't > guarantee any particular relationship between the two. Yes others can reduce the reservation, but the point of this is that the code doesn't care. We'll either see the value before or after the reduction and in either case we'll do something sensible. The worst that can happen is when we're just below the hard limit and multiple CPUs fail to allocate but that's not really a problem because if the machine is making progress at all then we will eventually scale back and allow these allocations to succeed. As to the non-atomic operation on memory_pressue, that's OK because we only ever assign values to it and never do other operations such as += or -=. Remember that int/long assignments must be atomic or Linux won't run on your architecture. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:05 ` Paul Mackerras 2007-08-16 2:11 ` Herbert Xu @ 2007-08-16 2:15 ` Christoph Lameter 2007-08-16 2:17 ` Christoph Lameter 2007-08-16 2:33 ` Satyam Sharma 2 siblings, 1 reply; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 2:15 UTC (permalink / raw) To: Paul Mackerras Cc: Herbert Xu, Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Paul Mackerras wrote: > > We don't need to reload sk->sk_prot->memory_allocated here. > > Are you sure? How do you know some other CPU hasn't changed the value > in between? The cpu knows because the cacheline was not invalidated. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:15 ` Christoph Lameter @ 2007-08-16 2:17 ` Christoph Lameter 0 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 2:17 UTC (permalink / raw) To: Paul Mackerras Cc: Herbert Xu, Paul E. McKenney, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Wed, 15 Aug 2007, Christoph Lameter wrote: > On Thu, 16 Aug 2007, Paul Mackerras wrote: > > > > We don't need to reload sk->sk_prot->memory_allocated here. > > > > Are you sure? How do you know some other CPU hasn't changed the value > > in between? > > The cpu knows because the cacheline was not invalidated. Crap my statement above is wrong..... We do not care that the value was changed otherwise we would have put a barrier in there. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:05 ` Paul Mackerras 2007-08-16 2:11 ` Herbert Xu 2007-08-16 2:15 ` Christoph Lameter @ 2007-08-16 2:33 ` Satyam Sharma 2007-08-16 3:01 ` Satyam Sharma 2007-08-16 3:05 ` Paul Mackerras 2 siblings, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 2:33 UTC (permalink / raw) To: Paul Mackerras Cc: Herbert Xu, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Paul Mackerras wrote: > Herbert Xu writes: > > > See sk_stream_mem_schedule in net/core/stream.c: > > > > /* Under limit. */ > > if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) { > > if (*sk->sk_prot->memory_pressure) > > *sk->sk_prot->memory_pressure = 0; > > return 1; > > } > > > > /* Over hard limit. */ > > if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) { > > sk->sk_prot->enter_memory_pressure(); > > goto suppress_allocation; > > } > > > > We don't need to reload sk->sk_prot->memory_allocated here. > > Are you sure? How do you know some other CPU hasn't changed the value > in between? I can't speak for this particular case, but there could be similar code examples elsewhere, where we do the atomic ops on an atomic_t object inside a higher-level locking scheme that would take care of the kind of problem you're referring to here. It would be useful for such or similar code if the compiler kept the value of that atomic object in a register. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:33 ` Satyam Sharma @ 2007-08-16 3:01 ` Satyam Sharma 2007-08-16 4:11 ` Paul Mackerras 2007-08-16 3:05 ` Paul Mackerras 1 sibling, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 3:01 UTC (permalink / raw) To: Paul Mackerras Cc: Herbert Xu, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Satyam Sharma wrote: > On Thu, 16 Aug 2007, Paul Mackerras wrote: > > Herbert Xu writes: > > > > > See sk_stream_mem_schedule in net/core/stream.c: > > > > > > /* Under limit. */ > > > if (atomic_read(sk->sk_prot->memory_allocated) < sk->sk_prot->sysctl_mem[0]) { > > > if (*sk->sk_prot->memory_pressure) > > > *sk->sk_prot->memory_pressure = 0; > > > return 1; > > > } > > > > > > /* Over hard limit. */ > > > if (atomic_read(sk->sk_prot->memory_allocated) > sk->sk_prot->sysctl_mem[2]) { > > > sk->sk_prot->enter_memory_pressure(); > > > goto suppress_allocation; > > > } > > > > > > We don't need to reload sk->sk_prot->memory_allocated here. > > > > Are you sure? How do you know some other CPU hasn't changed the value > > in between? > > I can't speak for this particular case, but there could be similar code > examples elsewhere, where we do the atomic ops on an atomic_t object > inside a higher-level locking scheme that would take care of the kind of > problem you're referring to here. It would be useful for such or similar > code if the compiler kept the value of that atomic object in a register. We might not be using atomic_t (and ops) if we already have a higher-level locking scheme, actually. So as Herbert mentioned, such cases might just not care. [ Too much of this thread, too little sleep, sorry! ] Anyway, the problem, of course, is that this conversion to a stronger / safer-by-default behaviour doesn't happen with zero cost to performance. Converting atomic ops to "volatile" behaviour did add ~2K to kernel text for archs such as i386 (possibly to important codepaths) that didn't have those semantics already so it would be constructive to actually look at those differences and see if there were really any heisenbugs that got rectified. Or if there were legitimate optimizations that got wrongly disabled. Onus lies on those proposing the modifications, I'd say ;-) ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:01 ` Satyam Sharma @ 2007-08-16 4:11 ` Paul Mackerras 2007-08-16 5:39 ` Herbert Xu 2007-08-16 18:54 ` Christoph Lameter 0 siblings, 2 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 4:11 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma writes: > Anyway, the problem, of course, is that this conversion to a stronger / > safer-by-default behaviour doesn't happen with zero cost to performance. > Converting atomic ops to "volatile" behaviour did add ~2K to kernel text > for archs such as i386 (possibly to important codepaths) that didn't have > those semantics already so it would be constructive to actually look at > those differences and see if there were really any heisenbugs that got > rectified. Or if there were legitimate optimizations that got wrongly > disabled. Onus lies on those proposing the modifications, I'd say ;-) The uses of atomic_read where one might want it to allow caching of the result seem to me to fall into 3 categories: 1. Places that are buggy because of a race arising from the way it's used. 2. Places where there is a race but it doesn't matter because we're doing some clever trick. 3. Places where there is some locking in place that eliminates any potential race. In case 1, adding volatile won't solve the race, of course, but it's hard to argue that we shouldn't do something because it will slow down buggy code. Case 2 is hopefully pretty rare and accompanied by large comment blocks, and in those cases caching the result of atomic_read explicitly in a local variable would probably make the code clearer. And in case 3 there is no reason to use atomic_t at all; we might as well just use an int. So I don't see any good reason to make the atomic API more complex by having "volatile" and "non-volatile" versions of atomic_read. It should just have the "volatile" behaviour. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 4:11 ` Paul Mackerras @ 2007-08-16 5:39 ` Herbert Xu 2007-08-16 6:56 ` Paul Mackerras 2007-08-16 18:54 ` Christoph Lameter 1 sibling, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 5:39 UTC (permalink / raw) To: Paul Mackerras Cc: Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 02:11:43PM +1000, Paul Mackerras wrote: > > The uses of atomic_read where one might want it to allow caching of > the result seem to me to fall into 3 categories: > > 1. Places that are buggy because of a race arising from the way it's > used. > > 2. Places where there is a race but it doesn't matter because we're > doing some clever trick. > > 3. Places where there is some locking in place that eliminates any > potential race. Agreed. > In case 1, adding volatile won't solve the race, of course, but it's > hard to argue that we shouldn't do something because it will slow down > buggy code. Case 2 is hopefully pretty rare and accompanied by large > comment blocks, and in those cases caching the result of atomic_read > explicitly in a local variable would probably make the code clearer. > And in case 3 there is no reason to use atomic_t at all; we might as > well just use an int. Since adding volatile doesn't help any of the 3 cases, and takes away optimisations from both 2 and 3, I wonder what is the point of the addition after all? Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 5:39 ` Herbert Xu @ 2007-08-16 6:56 ` Paul Mackerras 2007-08-16 7:09 ` Herbert Xu 0 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 6:56 UTC (permalink / raw) To: Herbert Xu Cc: Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > On Thu, Aug 16, 2007 at 02:11:43PM +1000, Paul Mackerras wrote: > > > > The uses of atomic_read where one might want it to allow caching of > > the result seem to me to fall into 3 categories: > > > > 1. Places that are buggy because of a race arising from the way it's > > used. > > > > 2. Places where there is a race but it doesn't matter because we're > > doing some clever trick. > > > > 3. Places where there is some locking in place that eliminates any > > potential race. > > Agreed. > > > In case 1, adding volatile won't solve the race, of course, but it's > > hard to argue that we shouldn't do something because it will slow down > > buggy code. Case 2 is hopefully pretty rare and accompanied by large > > comment blocks, and in those cases caching the result of atomic_read > > explicitly in a local variable would probably make the code clearer. > > And in case 3 there is no reason to use atomic_t at all; we might as > > well just use an int. > > Since adding volatile doesn't help any of the 3 cases, and > takes away optimisations from both 2 and 3, I wonder what > is the point of the addition after all? Note that I said these are the cases _where one might want to allow caching_, so of course adding volatile doesn't help _these_ cases. There are of course other cases where one definitely doesn't want to allow the compiler to cache the value, such as when polling an atomic variable waiting for another CPU to change it, and from my inspection so far these cases seem to be the majority. The reasons for having "volatile" behaviour of atomic_read (whether or not that is achieved by use of the "volatile" C keyword) are - It matches the normal expectation based on the name "atomic_read" - It matches the behaviour of the other atomic_* primitives - It avoids bugs in the cases where "volatile" behaviour is required To my mind these outweigh the small benefit for some code of the non-volatile (caching-allowed) behaviour. In fact it's pretty minor either way, and since x86[-64] has this behaviour, one can expect the potential bugs in generic code to have mostly been found, although perhaps not all of them since x86[-64] has less aggressive reordering of memory accesses and fewer registers in which to cache things than some other architectures. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 6:56 ` Paul Mackerras @ 2007-08-16 7:09 ` Herbert Xu 2007-08-16 8:06 ` Stefan Richter 2007-08-16 14:48 ` Ilpo Järvinen 0 siblings, 2 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-16 7:09 UTC (permalink / raw) To: Paul Mackerras Cc: Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 04:56:21PM +1000, Paul Mackerras wrote: > > Note that I said these are the cases _where one might want to allow > caching_, so of course adding volatile doesn't help _these_ cases. > There are of course other cases where one definitely doesn't want to > allow the compiler to cache the value, such as when polling an atomic > variable waiting for another CPU to change it, and from my inspection > so far these cases seem to be the majority. We've been through that already. If it's a busy-wait it should use cpu_relax. If it's scheduling away that already forces the compiler to reread anyway. Do you have an actual example where volatile is needed? > - It matches the normal expectation based on the name "atomic_read" > - It matches the behaviour of the other atomic_* primitives Can't argue since you left out what those expectations or properties are. > - It avoids bugs in the cases where "volatile" behaviour is required Do you (or anyone else for that matter) have an example of this? Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 7:09 ` Herbert Xu @ 2007-08-16 8:06 ` Stefan Richter 2007-08-16 8:10 ` Herbert Xu 2007-08-16 14:48 ` Ilpo Järvinen 1 sibling, 1 reply; 333+ messages in thread From: Stefan Richter @ 2007-08-16 8:06 UTC (permalink / raw) To: Herbert Xu Cc: Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > On Thu, Aug 16, 2007 at 04:56:21PM +1000, Paul Mackerras wrote: >> >> Note that I said these are the cases _where one might want to allow >> caching_, so of course adding volatile doesn't help _these_ cases. >> There are of course other cases where one definitely doesn't want to >> allow the compiler to cache the value, such as when polling an atomic >> variable waiting for another CPU to change it, and from my inspection >> so far these cases seem to be the majority. > > We've been through that already. If it's a busy-wait it > should use cpu_relax. If it's scheduling away that already > forces the compiler to reread anyway. > > Do you have an actual example where volatile is needed? > >> - It matches the normal expectation based on the name "atomic_read" >> - It matches the behaviour of the other atomic_* primitives > > Can't argue since you left out what those expectations > or properties are. We use atomic_t for data that is concurrently locklessly written and read at arbitrary times. My naive expectation as driver author (driver maintainer) is that all atomic_t accessors, including atomic_read, (and atomic bitops) work with the then current value of the atomic data. >> - It avoids bugs in the cases where "volatile" behaviour is required > > Do you (or anyone else for that matter) have an example of this? The only code I somewhat know, the ieee1394 subsystem, was perhaps authored and is currently maintained with the expectation that each occurrence of atomic_read actually results in a load operation, i.e. is not optimized away. This means all atomic_t (bus generation, packet and buffer refcounts, and some other state variables)* and likewise all atomic bitops in that subsystem. If that assumption is wrong, then what is the API or language primitive to force a load operation to occur? *) Interesting what a quick LXR session in search for all atomic_t usages in 'my' subsystem brings to light. I now noticed an apparently unused struct member in the bitrotting pcilynx driver, and more importantly, a pairing of two atomic_t variables in raw1394 that should be audited for race conditions and for possible replacement by plain int. -- Stefan Richter -=====-=-=== =--- =---- http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 8:06 ` Stefan Richter @ 2007-08-16 8:10 ` Herbert Xu 2007-08-16 9:54 ` Stefan Richter ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-16 8:10 UTC (permalink / raw) To: Stefan Richter Cc: Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote: > > > > Do you (or anyone else for that matter) have an example of this? > > The only code I somewhat know, the ieee1394 subsystem, was perhaps > authored and is currently maintained with the expectation that each > occurrence of atomic_read actually results in a load operation, i.e. is > not optimized away. This means all atomic_t (bus generation, packet and > buffer refcounts, and some other state variables)* and likewise all > atomic bitops in that subsystem. Can you find an actual atomic_read code snippet there that is broken without the volatile modifier? Thanks, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 8:10 ` Herbert Xu @ 2007-08-16 9:54 ` Stefan Richter 2007-08-16 10:31 ` Stefan Richter 2007-08-16 10:35 ` Herbert Xu 2007-08-16 19:48 ` Chris Snook 2007-08-17 5:09 ` Paul Mackerras 2 siblings, 2 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-16 9:54 UTC (permalink / raw) To: Herbert Xu Cc: Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote: >> > >> > Do you (or anyone else for that matter) have an example of this? >> >> The only code I somewhat know, the ieee1394 subsystem, was perhaps >> authored and is currently maintained with the expectation that each >> occurrence of atomic_read actually results in a load operation, i.e. is >> not optimized away. This means all atomic_t (bus generation, packet and >> buffer refcounts, and some other state variables)* and likewise all >> atomic bitops in that subsystem. > > Can you find an actual atomic_read code snippet there that is > broken without the volatile modifier? What do I have to look for? atomic_read after another read or write access to the same variable, in the same function scope? Or in the sum of scopes of functions that could be merged by function inlining? One example was discussed here earlier: The for (;;) loop in nodemgr_host_thread. There an msleep_interruptible implicitly acted as barrier (at the moment because it's in a different translation unit; if it were the same, then because it hopefully has own barriers). So that happens to work, although such an implicit barrier is bad style: Better enforce the desired behaviour (== guaranteed load operation) *explicitly*. -- Stefan Richter -=====-=-=== =--- =---- http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 9:54 ` Stefan Richter @ 2007-08-16 10:31 ` Stefan Richter 2007-08-16 10:42 ` Herbert Xu 2007-08-16 10:35 ` Herbert Xu 1 sibling, 1 reply; 333+ messages in thread From: Stefan Richter @ 2007-08-16 10:31 UTC (permalink / raw) To: Herbert Xu Cc: Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher I wrote: > Herbert Xu wrote: >> On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote: [...] >>> expectation that each >>> occurrence of atomic_read actually results in a load operation, i.e. is >>> not optimized away. [...] >> Can you find an actual atomic_read code snippet there that is >> broken without the volatile modifier? PS: Just to clarify, I'm not speaking for the volatile modifier. I'm not speaking for any particular implementation of atomic_t and its accessors at all. All I am saying is that - we use atomically accessed data types because we concurrently but locklessly access this data, - hence a read access to this data that could be optimized away makes *no sense at all*. The only sensible read accessor to an atomic datatype is a read accessor that will not be optimized away. So, the architecture guys can implement atomic_read however they want --- as long as it cannot be optimized away.* PPS: If somebody has code where he can afford to let the compiler coalesce atomic_read with a previous access to the same data, i.e. doesn't need and doesn't want all guarantees that the atomic_read API makes (or IMO should make), then he can replace the atomic_read by a local temporary variable. *) Exceptions: if (known_to_be_false) read_access(a); and the like. -- Stefan Richter -=====-=-=== =--- =---- http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 10:31 ` Stefan Richter @ 2007-08-16 10:42 ` Herbert Xu 2007-08-16 16:34 ` Paul E. McKenney 2007-08-17 5:04 ` Paul Mackerras 0 siblings, 2 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-16 10:42 UTC (permalink / raw) To: Stefan Richter Cc: Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 12:31:03PM +0200, Stefan Richter wrote: > > PS: Just to clarify, I'm not speaking for the volatile modifier. I'm > not speaking for any particular implementation of atomic_t and its > accessors at all. All I am saying is that > - we use atomically accessed data types because we concurrently but > locklessly access this data, > - hence a read access to this data that could be optimized away > makes *no sense at all*. No sane compiler can optimise away an atomic_read per se. That's only possible if there's a preceding atomic_set or atomic_read, with no barriers in the middle. If that's the case, then one has to conclude that doing away with the second read is acceptable, as otherwise a memory (or at least a compiler) barrier should have been used. In fact, volatile doesn't guarantee that the memory gets read anyway. You might be reading some stale value out of the cache. Granted this doesn't happen on x86 but when you're coding for the kernel you can't make such assumptions. So the point here is that if you don't mind getting a stale value from the CPU cache when doing an atomic_read, then surely you won't mind getting a stale value from the compiler "cache". > So, the architecture guys can implement atomic_read however they want > --- as long as it cannot be optimized away.* They can implement it however they want as long as it stays atomic. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 10:42 ` Herbert Xu @ 2007-08-16 16:34 ` Paul E. McKenney 2007-08-16 23:59 ` Herbert Xu 2007-08-17 3:15 ` Nick Piggin 2007-08-17 5:04 ` Paul Mackerras 1 sibling, 2 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 16:34 UTC (permalink / raw) To: Herbert Xu Cc: Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 06:42:50PM +0800, Herbert Xu wrote: > On Thu, Aug 16, 2007 at 12:31:03PM +0200, Stefan Richter wrote: > > > > PS: Just to clarify, I'm not speaking for the volatile modifier. I'm > > not speaking for any particular implementation of atomic_t and its > > accessors at all. All I am saying is that > > - we use atomically accessed data types because we concurrently but > > locklessly access this data, > > - hence a read access to this data that could be optimized away > > makes *no sense at all*. > > No sane compiler can optimise away an atomic_read per se. > That's only possible if there's a preceding atomic_set or > atomic_read, with no barriers in the middle. > > If that's the case, then one has to conclude that doing > away with the second read is acceptable, as otherwise > a memory (or at least a compiler) barrier should have been > used. The compiler can also reorder non-volatile accesses. For an example patch that cares about this, please see: http://lkml.org/lkml/2007/8/7/280 This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and rcu_read_unlock() to ensure that accesses aren't reordered with respect to interrupt handlers and NMIs/SMIs running on that same CPU. > In fact, volatile doesn't guarantee that the memory gets > read anyway. You might be reading some stale value out > of the cache. Granted this doesn't happen on x86 but > when you're coding for the kernel you can't make such > assumptions. > > So the point here is that if you don't mind getting a stale > value from the CPU cache when doing an atomic_read, then > surely you won't mind getting a stale value from the compiler > "cache". Absolutely disagree. An interrupt/NMI/SMI handler running on the CPU will see the same value (whether in cache or in store buffer) that the mainline code will see. In this case, we don't care about CPU misordering, only about compiler misordering. It is easy to see other uses that combine communication with handlers on the current CPU with communication among CPUs -- again, see prior messages in this thread. > > So, the architecture guys can implement atomic_read however they want > > --- as long as it cannot be optimized away.* > > They can implement it however they want as long as it stays > atomic. Precisely. And volatility is a key property of "atomic". Let's please not throw it away. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 16:34 ` Paul E. McKenney @ 2007-08-16 23:59 ` Herbert Xu 2007-08-17 1:01 ` Paul E. McKenney 2007-08-17 3:15 ` Nick Piggin 1 sibling, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-16 23:59 UTC (permalink / raw) To: Paul E. McKenney Cc: Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote: > > The compiler can also reorder non-volatile accesses. For an example > patch that cares about this, please see: > > http://lkml.org/lkml/2007/8/7/280 > > This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and > rcu_read_unlock() to ensure that accesses aren't reordered with respect > to interrupt handlers and NMIs/SMIs running on that same CPU. Good, finally we have some code to discuss (even though it's not actually in the kernel yet). First of all, I think this illustrates that what you want here has nothing to do with atomic ops. The ORDERED_WRT_IRQ macro occurs a lot more times in your patch than atomic reads/sets. So *assuming* that it was necessary at all, then having an ordered variant of the atomic_read/atomic_set ops could do just as well. However, I still don't know which atomic_read/atomic_set in your patch would be broken if there were no volatile. Could you please point them out? Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 23:59 ` Herbert Xu @ 2007-08-17 1:01 ` Paul E. McKenney 2007-08-17 7:39 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 1:01 UTC (permalink / raw) To: Herbert Xu Cc: Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote: > On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote: > > > > The compiler can also reorder non-volatile accesses. For an example > > patch that cares about this, please see: > > > > http://lkml.org/lkml/2007/8/7/280 > > > > This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and > > rcu_read_unlock() to ensure that accesses aren't reordered with respect > > to interrupt handlers and NMIs/SMIs running on that same CPU. > > Good, finally we have some code to discuss (even though it's > not actually in the kernel yet). There was some earlier in this thread as well. > First of all, I think this illustrates that what you want > here has nothing to do with atomic ops. The ORDERED_WRT_IRQ > macro occurs a lot more times in your patch than atomic > reads/sets. So *assuming* that it was necessary at all, > then having an ordered variant of the atomic_read/atomic_set > ops could do just as well. Indeed. If I could trust atomic_read()/atomic_set() to cause the compiler to maintain ordering, then I could just use them instead of having to create an ORDERED_WRT_IRQ(). (Or ACCESS_ONCE(), as it is called in a different patch.) > However, I still don't know which atomic_read/atomic_set in > your patch would be broken if there were no volatile. Could > you please point them out? Suppose I tried replacing the ORDERED_WRT_IRQ() calls with atomic_read() and atomic_set(). Starting with __rcu_read_lock(): o If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++" was ordered by the compiler after "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then suppose an NMI/SMI happened after the rcu_read_lock_nesting but before the rcu_flipctr. Then if there was an rcu_read_lock() in the SMI/NMI handler (which is perfectly legal), the nested rcu_read_lock() would believe that it could take the then-clause of the enclosing "if" statement. But because the rcu_flipctr per-CPU variable had not yet been incremented, an RCU updater would be within its rights to assume that there were no RCU reads in progress, thus possibly yanking a data structure out from under the reader in the SMI/NMI function. Fatal outcome. Note that only one CPU is involved here because these are all either per-CPU or per-task variables. o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1" was ordered by the compiler to follow the "ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI happened between the two, then an __rcu_read_lock() in the NMI/SMI would incorrectly take the "else" clause of the enclosing "if" statement. If some other CPU flipped the rcu_ctrlblk.completed in the meantime, then the __rcu_read_lock() would (correctly) write the new value into rcu_flipctr_idx. Well and good so far. But the problem arises in __rcu_read_unlock(), which then decrements the wrong counter. Depending on exactly how subsequent events played out, this could result in either prematurely ending grace periods or never-ending grace periods, both of which are fatal outcomes. And the following are not needed in the current version of the patch, but will be in a future version that either avoids disabling irqs or that dispenses with the smp_read_barrier_depends() that I have 99% convinced myself is unneeded: o nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting); o idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1; Furthermore, in that future version, irq handlers can cause the same mischief that SMI/NMI handlers can in this version. Next, looking at __rcu_read_unlock(): o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1" was reordered by the compiler to follow the "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--", then if an NMI/SMI containing an rcu_read_lock() occurs between the two, this nested rcu_read_lock() would incorrectly believe that it was protected by an enclosing RCU read-side critical section as described in the first reversal discussed for __rcu_read_lock() above. Again, fatal outcome. This is what we have now. It is not hard to imagine situations that interact with -both- interrupt handlers -and- other CPUs, as described earlier. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 1:01 ` Paul E. McKenney @ 2007-08-17 7:39 ` Satyam Sharma 2007-08-17 14:31 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 7:39 UTC (permalink / raw) To: Paul E. McKenney Cc: Herbert Xu, Stefan Richter, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Paul E. McKenney wrote: > On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote: > > On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote: > > > > > > The compiler can also reorder non-volatile accesses. For an example > > > patch that cares about this, please see: > > > > > > http://lkml.org/lkml/2007/8/7/280 > > > > > > This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and > > > rcu_read_unlock() to ensure that accesses aren't reordered with respect > > > to interrupt handlers and NMIs/SMIs running on that same CPU. > > > > Good, finally we have some code to discuss (even though it's > > not actually in the kernel yet). > > There was some earlier in this thread as well. Hmm, I never quite got what all this interrupt/NMI/SMI handling and RCU business you mentioned earlier was all about, but now that you've pointed to the actual code and issues with it ... > > First of all, I think this illustrates that what you want > > here has nothing to do with atomic ops. The ORDERED_WRT_IRQ > > macro occurs a lot more times in your patch than atomic > > reads/sets. So *assuming* that it was necessary at all, > > then having an ordered variant of the atomic_read/atomic_set > > ops could do just as well. > > Indeed. If I could trust atomic_read()/atomic_set() to cause the compiler > to maintain ordering, then I could just use them instead of having to > create an ORDERED_WRT_IRQ(). (Or ACCESS_ONCE(), as it is called in a > different patch.) +#define WHATEVER(x) (*(volatile typeof(x) *)&(x)) I suppose one could want volatile access semantics for stuff that's a bit-field too, no? Also, this gives *zero* "re-ordering" guarantees that your code wants as you've explained it below) -- neither w.r.t. CPU re-ordering (which probably you don't care about) *nor* w.r.t. compiler re-ordering (which you definitely _do_ care about). > > However, I still don't know which atomic_read/atomic_set in > > your patch would be broken if there were no volatile. Could > > you please point them out? > > Suppose I tried replacing the ORDERED_WRT_IRQ() calls with > atomic_read() and atomic_set(). Starting with __rcu_read_lock(): > > o If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++" > was ordered by the compiler after > "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then > suppose an NMI/SMI happened after the rcu_read_lock_nesting but > before the rcu_flipctr. > > Then if there was an rcu_read_lock() in the SMI/NMI > handler (which is perfectly legal), the nested rcu_read_lock() > would believe that it could take the then-clause of the > enclosing "if" statement. But because the rcu_flipctr per-CPU > variable had not yet been incremented, an RCU updater would > be within its rights to assume that there were no RCU reads > in progress, thus possibly yanking a data structure out from > under the reader in the SMI/NMI function. > > Fatal outcome. Note that only one CPU is involved here > because these are all either per-CPU or per-task variables. Ok, so you don't care about CPU re-ordering. Still, I should let you know that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you want is a full compiler optimization barrier(). [ Your code probably works now, and emits correct code, but that's just because of gcc did what it did. Nothing in any standard, or in any documented behaviour of gcc, or anything about the real (or expected) semantics of "volatile" is protecting the code here. ] > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1" > was ordered by the compiler to follow the > "ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI > happened between the two, then an __rcu_read_lock() in the NMI/SMI > would incorrectly take the "else" clause of the enclosing "if" > statement. If some other CPU flipped the rcu_ctrlblk.completed > in the meantime, then the __rcu_read_lock() would (correctly) > write the new value into rcu_flipctr_idx. > > Well and good so far. But the problem arises in > __rcu_read_unlock(), which then decrements the wrong counter. > Depending on exactly how subsequent events played out, this could > result in either prematurely ending grace periods or never-ending > grace periods, both of which are fatal outcomes. > > And the following are not needed in the current version of the > patch, but will be in a future version that either avoids disabling > irqs or that dispenses with the smp_read_barrier_depends() that I > have 99% convinced myself is unneeded: > > o nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting); > > o idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1; > > Furthermore, in that future version, irq handlers can cause the same > mischief that SMI/NMI handlers can in this version. > > Next, looking at __rcu_read_unlock(): > > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1" > was reordered by the compiler to follow the > "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--", > then if an NMI/SMI containing an rcu_read_lock() occurs between > the two, this nested rcu_read_lock() would incorrectly believe > that it was protected by an enclosing RCU read-side critical > section as described in the first reversal discussed for > __rcu_read_lock() above. Again, fatal outcome. > > This is what we have now. It is not hard to imagine situations that > interact with -both- interrupt handlers -and- other CPUs, as described > earlier. It's not about interrupt/SMI/NMI handlers at all! What you clearly want, simply put, is that a certain stream of C statements must be emitted by the compiler _as they are_ with no re-ordering optimizations! You must *definitely* use barrier(), IMHO. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 7:39 ` Satyam Sharma @ 2007-08-17 14:31 ` Paul E. McKenney 2007-08-17 18:31 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 14:31 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Stefan Richter, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, Aug 17, 2007 at 01:09:08PM +0530, Satyam Sharma wrote: > > > On Thu, 16 Aug 2007, Paul E. McKenney wrote: > > > On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote: > > > On Thu, Aug 16, 2007 at 09:34:41AM -0700, Paul E. McKenney wrote: > > > > > > > > The compiler can also reorder non-volatile accesses. For an example > > > > patch that cares about this, please see: > > > > > > > > http://lkml.org/lkml/2007/8/7/280 > > > > > > > > This patch uses an ORDERED_WRT_IRQ() in rcu_read_lock() and > > > > rcu_read_unlock() to ensure that accesses aren't reordered with respect > > > > to interrupt handlers and NMIs/SMIs running on that same CPU. > > > > > > Good, finally we have some code to discuss (even though it's > > > not actually in the kernel yet). > > > > There was some earlier in this thread as well. > > Hmm, I never quite got what all this interrupt/NMI/SMI handling and > RCU business you mentioned earlier was all about, but now that you've > pointed to the actual code and issues with it ... Glad to help... > > > First of all, I think this illustrates that what you want > > > here has nothing to do with atomic ops. The ORDERED_WRT_IRQ > > > macro occurs a lot more times in your patch than atomic > > > reads/sets. So *assuming* that it was necessary at all, > > > then having an ordered variant of the atomic_read/atomic_set > > > ops could do just as well. > > > > Indeed. If I could trust atomic_read()/atomic_set() to cause the compiler > > to maintain ordering, then I could just use them instead of having to > > create an ORDERED_WRT_IRQ(). (Or ACCESS_ONCE(), as it is called in a > > different patch.) > > +#define WHATEVER(x) (*(volatile typeof(x) *)&(x)) > > I suppose one could want volatile access semantics for stuff that's > a bit-field too, no? One could, but this is not supported in general. So if you want that, you need to use the usual bit-mask tricks and (for setting) atomic operations. > Also, this gives *zero* "re-ordering" guarantees that your code wants > as you've explained it below) -- neither w.r.t. CPU re-ordering (which > probably you don't care about) *nor* w.r.t. compiler re-ordering > (which you definitely _do_ care about). You are correct about CPU re-ordering (and about the fact that this example doesn't care about it), but not about compiler re-ordering. The compiler is prohibited from moving a volatile access across a sequence point. One example of a sequence point is a statement boundary. Because all of the volatile accesses in this code are separated by statement boundaries, a conforming compiler is prohibited from reordering them. > > > However, I still don't know which atomic_read/atomic_set in > > > your patch would be broken if there were no volatile. Could > > > you please point them out? > > > > Suppose I tried replacing the ORDERED_WRT_IRQ() calls with > > atomic_read() and atomic_set(). Starting with __rcu_read_lock(): > > > > o If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++" > > was ordered by the compiler after > > "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then > > suppose an NMI/SMI happened after the rcu_read_lock_nesting but > > before the rcu_flipctr. > > > > Then if there was an rcu_read_lock() in the SMI/NMI > > handler (which is perfectly legal), the nested rcu_read_lock() > > would believe that it could take the then-clause of the > > enclosing "if" statement. But because the rcu_flipctr per-CPU > > variable had not yet been incremented, an RCU updater would > > be within its rights to assume that there were no RCU reads > > in progress, thus possibly yanking a data structure out from > > under the reader in the SMI/NMI function. > > > > Fatal outcome. Note that only one CPU is involved here > > because these are all either per-CPU or per-task variables. > > Ok, so you don't care about CPU re-ordering. Still, I should let you know > that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you > want is a full compiler optimization barrier(). No. See above. > [ Your code probably works now, and emits correct code, but that's > just because of gcc did what it did. Nothing in any standard, > or in any documented behaviour of gcc, or anything about the real > (or expected) semantics of "volatile" is protecting the code here. ] Really? Why doesn't the prohibition against moving volatile accesses across sequence points take care of this? > > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1" > > was ordered by the compiler to follow the > > "ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI > > happened between the two, then an __rcu_read_lock() in the NMI/SMI > > would incorrectly take the "else" clause of the enclosing "if" > > statement. If some other CPU flipped the rcu_ctrlblk.completed > > in the meantime, then the __rcu_read_lock() would (correctly) > > write the new value into rcu_flipctr_idx. > > > > Well and good so far. But the problem arises in > > __rcu_read_unlock(), which then decrements the wrong counter. > > Depending on exactly how subsequent events played out, this could > > result in either prematurely ending grace periods or never-ending > > grace periods, both of which are fatal outcomes. > > > > And the following are not needed in the current version of the > > patch, but will be in a future version that either avoids disabling > > irqs or that dispenses with the smp_read_barrier_depends() that I > > have 99% convinced myself is unneeded: > > > > o nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting); > > > > o idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1; > > > > Furthermore, in that future version, irq handlers can cause the same > > mischief that SMI/NMI handlers can in this version. > > > > Next, looking at __rcu_read_unlock(): > > > > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1" > > was reordered by the compiler to follow the > > "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--", > > then if an NMI/SMI containing an rcu_read_lock() occurs between > > the two, this nested rcu_read_lock() would incorrectly believe > > that it was protected by an enclosing RCU read-side critical > > section as described in the first reversal discussed for > > __rcu_read_lock() above. Again, fatal outcome. > > > > This is what we have now. It is not hard to imagine situations that > > interact with -both- interrupt handlers -and- other CPUs, as described > > earlier. > > It's not about interrupt/SMI/NMI handlers at all! What you clearly want, > simply put, is that a certain stream of C statements must be emitted > by the compiler _as they are_ with no re-ordering optimizations! You must > *definitely* use barrier(), IMHO. Almost. I don't care about most of the operations, only about the loads and stores marked volatile. Again, although the compiler is free to reorder volatile accesses that occur -within- a single statement, it is prohibited by the standard from moving volatile accesses from one statement to another. Therefore, this code can legitimately use volatile. Or am I missing something subtle? Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 14:31 ` Paul E. McKenney @ 2007-08-17 18:31 ` Satyam Sharma 2007-08-17 18:56 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 18:31 UTC (permalink / raw) To: Paul E. McKenney Cc: Herbert Xu, Stefan Richter, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Paul E. McKenney wrote: > On Fri, Aug 17, 2007 at 01:09:08PM +0530, Satyam Sharma wrote: > > > > On Thu, 16 Aug 2007, Paul E. McKenney wrote: > > > > > On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote: > > > > > > > > First of all, I think this illustrates that what you want > > > > here has nothing to do with atomic ops. The ORDERED_WRT_IRQ > > > > macro occurs a lot more times in your patch than atomic > > > > reads/sets. So *assuming* that it was necessary at all, > > > > then having an ordered variant of the atomic_read/atomic_set > > > > ops could do just as well. > > > > > > Indeed. If I could trust atomic_read()/atomic_set() to cause the compiler > > > to maintain ordering, then I could just use them instead of having to > > > create an ORDERED_WRT_IRQ(). (Or ACCESS_ONCE(), as it is called in a > > > different patch.) > > > > +#define WHATEVER(x) (*(volatile typeof(x) *)&(x)) > > [...] > > Also, this gives *zero* "re-ordering" guarantees that your code wants > > as you've explained it below) -- neither w.r.t. CPU re-ordering (which > > probably you don't care about) *nor* w.r.t. compiler re-ordering > > (which you definitely _do_ care about). > > You are correct about CPU re-ordering (and about the fact that this > example doesn't care about it), but not about compiler re-ordering. > > The compiler is prohibited from moving a volatile access across a sequence > point. One example of a sequence point is a statement boundary. Because > all of the volatile accesses in this code are separated by statement > boundaries, a conforming compiler is prohibited from reordering them. Yes, you're right, and I believe precisely this was discussed elsewhere as well today. But I'd call attention to what Herbert mentioned there. You're using ORDERED_WRT_IRQ() on stuff that is _not_ defined to be an atomic_t at all: * Member "completed" of struct rcu_ctrlblk is a long. * Per-cpu variable rcu_flipctr is an array of ints. * Members "rcu_read_lock_nesting" and "rcu_flipctr_idx" of struct task_struct are ints. So are you saying you're "having to use" this volatile-access macro because you *couldn't* declare all the above as atomic_t and thus just expect the right thing to happen by using the atomic ops API by default, because it lacks volatile access semantics (on x86)? If so, then I wonder if using the volatile access cast is really the best way to achieve (at least in terms of code clarity) the kind of re-ordering guarantees it wants there. (there could be alternative solutions, such as using barrier(), or that at bottom of this mail) What I mean is this: If you switch to atomic_t, and x86 switched to make atomic_t have "volatile" semantics by default, the statements would be simply a string of: atomic_inc(), atomic_add(), atomic_set(), and atomic_read() statements, and nothing in there that clearly makes it *explicit* that the code is correct (and not buggy) simply because of the re-ordering guarantees that the C "volatile" type-qualifier keyword gives us as per the standard. But now we're firmly in "subjective" territory, so you or anybody could legitimately disagree. > > > Suppose I tried replacing the ORDERED_WRT_IRQ() calls with > > > atomic_read() and atomic_set(). Starting with __rcu_read_lock(): > > > > > > o If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++" > > > was ordered by the compiler after > > > "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then > > > suppose an NMI/SMI happened after the rcu_read_lock_nesting but > > > before the rcu_flipctr. > > > > > > Then if there was an rcu_read_lock() in the SMI/NMI > > > handler (which is perfectly legal), the nested rcu_read_lock() > > > would believe that it could take the then-clause of the > > > enclosing "if" statement. But because the rcu_flipctr per-CPU > > > variable had not yet been incremented, an RCU updater would > > > be within its rights to assume that there were no RCU reads > > > in progress, thus possibly yanking a data structure out from > > > under the reader in the SMI/NMI function. > > > > > > Fatal outcome. Note that only one CPU is involved here > > > because these are all either per-CPU or per-task variables. > > > > Ok, so you don't care about CPU re-ordering. Still, I should let you know > > that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you > > want is a full compiler optimization barrier(). > > No. See above. True, *(volatile foo *)& _will_ work for this case. But multiple calls to barrier() (granted, would invalidate all other optimizations also) would work as well, would it not? [ Interestingly, if you declared all those objects mentioned earlier as atomic_t, and x86(-64) switched to an __asm__ __volatile__ based variant for atomic_{read,set}_volatile(), the bugs you want to avoid would still be there. "volatile" the C language type-qualifier does have compiler re-ordering semantics you mentioned earlier, but the "volatile" that applies to inline asm()s gives no re-ordering guarantees. ] > > > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1" > > > was ordered by the compiler to follow the > > > "ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI > > > happened between the two, then an __rcu_read_lock() in the NMI/SMI > > > would incorrectly take the "else" clause of the enclosing "if" > > > statement. If some other CPU flipped the rcu_ctrlblk.completed > > > in the meantime, then the __rcu_read_lock() would (correctly) > > > write the new value into rcu_flipctr_idx. > > > > > > Well and good so far. But the problem arises in > > > __rcu_read_unlock(), which then decrements the wrong counter. > > > Depending on exactly how subsequent events played out, this could > > > result in either prematurely ending grace periods or never-ending > > > grace periods, both of which are fatal outcomes. > > > > > > And the following are not needed in the current version of the > > > patch, but will be in a future version that either avoids disabling > > > irqs or that dispenses with the smp_read_barrier_depends() that I > > > have 99% convinced myself is unneeded: > > > > > > o nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting); > > > > > > o idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1; > > > > > > Furthermore, in that future version, irq handlers can cause the same > > > mischief that SMI/NMI handlers can in this version. So don't remove the local_irq_save/restore, which is well-established and well-understood for such cases (it doesn't help you with SMI/NMI, admittedly). This isn't really about RCU or per-cpu vars as such, it's just about racy code where you don't want to get hit by a concurrent interrupt (it does turn out that doing things in a _particular order_ will not cause fatal/buggy behaviour, but it's still a race issue, after all). > > > Next, looking at __rcu_read_unlock(): > > > > > > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1" > > > was reordered by the compiler to follow the > > > "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--", > > > then if an NMI/SMI containing an rcu_read_lock() occurs between > > > the two, this nested rcu_read_lock() would incorrectly believe > > > that it was protected by an enclosing RCU read-side critical > > > section as described in the first reversal discussed for > > > __rcu_read_lock() above. Again, fatal outcome. > > > > > > This is what we have now. It is not hard to imagine situations that > > > interact with -both- interrupt handlers -and- other CPUs, as described > > > earlier. Unless somebody's going for a lockless implementation, such situations normally use spin_lock_irqsave() based locking (or local_irq_save for those who care only for current CPU) -- problem with the patch in question, is that you want to prevent races with concurrent SMI/NMIs as well, which is not something that a lot of code needs to consider. [ Curiously, another thread is discussing something similar also: http://lkml.org/lkml/2007/8/15/393 "RFC: do get_rtc_time() correctly" ] Anyway, I didn't look at the code in that patch very much in detail, but why couldn't you implement some kind of synchronization variable that lets rcu_read_lock() or rcu_read_unlock() -- when being called from inside an NMI or SMI handler -- know that it has concurrently interrupted an ongoing rcu_read_{un}lock() and so must do things differently ... (?) I'm also wondering if there's other code that's not using locking in the kernel that faces similar issues, and what they've done to deal with it (if anything). Such bugs would be subtle, and difficult to diagnose. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 18:31 ` Satyam Sharma @ 2007-08-17 18:56 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 18:56 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Stefan Richter, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Sat, Aug 18, 2007 at 12:01:38AM +0530, Satyam Sharma wrote: > > > On Fri, 17 Aug 2007, Paul E. McKenney wrote: > > > On Fri, Aug 17, 2007 at 01:09:08PM +0530, Satyam Sharma wrote: > > > > > > On Thu, 16 Aug 2007, Paul E. McKenney wrote: > > > > > > > On Fri, Aug 17, 2007 at 07:59:02AM +0800, Herbert Xu wrote: > > > > > > > > > > First of all, I think this illustrates that what you want > > > > > here has nothing to do with atomic ops. The ORDERED_WRT_IRQ > > > > > macro occurs a lot more times in your patch than atomic > > > > > reads/sets. So *assuming* that it was necessary at all, > > > > > then having an ordered variant of the atomic_read/atomic_set > > > > > ops could do just as well. > > > > > > > > Indeed. If I could trust atomic_read()/atomic_set() to cause the compiler > > > > to maintain ordering, then I could just use them instead of having to > > > > create an ORDERED_WRT_IRQ(). (Or ACCESS_ONCE(), as it is called in a > > > > different patch.) > > > > > > +#define WHATEVER(x) (*(volatile typeof(x) *)&(x)) > > > [...] > > > Also, this gives *zero* "re-ordering" guarantees that your code wants > > > as you've explained it below) -- neither w.r.t. CPU re-ordering (which > > > probably you don't care about) *nor* w.r.t. compiler re-ordering > > > (which you definitely _do_ care about). > > > > You are correct about CPU re-ordering (and about the fact that this > > example doesn't care about it), but not about compiler re-ordering. > > > > The compiler is prohibited from moving a volatile access across a sequence > > point. One example of a sequence point is a statement boundary. Because > > all of the volatile accesses in this code are separated by statement > > boundaries, a conforming compiler is prohibited from reordering them. > > Yes, you're right, and I believe precisely this was discussed elsewhere > as well today. > > But I'd call attention to what Herbert mentioned there. You're using > ORDERED_WRT_IRQ() on stuff that is _not_ defined to be an atomic_t at all: > > * Member "completed" of struct rcu_ctrlblk is a long. > * Per-cpu variable rcu_flipctr is an array of ints. > * Members "rcu_read_lock_nesting" and "rcu_flipctr_idx" of > struct task_struct are ints. > > So are you saying you're "having to use" this volatile-access macro > because you *couldn't* declare all the above as atomic_t and thus just > expect the right thing to happen by using the atomic ops API by default, > because it lacks volatile access semantics (on x86)? > > If so, then I wonder if using the volatile access cast is really the > best way to achieve (at least in terms of code clarity) the kind of > re-ordering guarantees it wants there. (there could be alternative > solutions, such as using barrier(), or that at bottom of this mail) > > What I mean is this: If you switch to atomic_t, and x86 switched to > make atomic_t have "volatile" semantics by default, the statements > would be simply a string of: atomic_inc(), atomic_add(), atomic_set(), > and atomic_read() statements, and nothing in there that clearly makes > it *explicit* that the code is correct (and not buggy) simply because > of the re-ordering guarantees that the C "volatile" type-qualifier > keyword gives us as per the standard. But now we're firmly in > "subjective" territory, so you or anybody could legitimately disagree. In any case, given Linus's note, it appears that atomic_read() and atomic_set() won't consistently have volatile semantics, at least not while the compiler generates such ugly code for volatile accesses. So I will continue with my current approach. In any case, I will not be using atomic_inc() or atomic_add() in this code, as doing so would more than double the overhead, even on machines that are the most efficient at implementing atomic operations. > > > > Suppose I tried replacing the ORDERED_WRT_IRQ() calls with > > > > atomic_read() and atomic_set(). Starting with __rcu_read_lock(): > > > > > > > > o If "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])++" > > > > was ordered by the compiler after > > > > "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1", then > > > > suppose an NMI/SMI happened after the rcu_read_lock_nesting but > > > > before the rcu_flipctr. > > > > > > > > Then if there was an rcu_read_lock() in the SMI/NMI > > > > handler (which is perfectly legal), the nested rcu_read_lock() > > > > would believe that it could take the then-clause of the > > > > enclosing "if" statement. But because the rcu_flipctr per-CPU > > > > variable had not yet been incremented, an RCU updater would > > > > be within its rights to assume that there were no RCU reads > > > > in progress, thus possibly yanking a data structure out from > > > > under the reader in the SMI/NMI function. > > > > > > > > Fatal outcome. Note that only one CPU is involved here > > > > because these are all either per-CPU or per-task variables. > > > > > > Ok, so you don't care about CPU re-ordering. Still, I should let you know > > > that your ORDERED_WRT_IRQ() -- bad name, btw -- is still buggy. What you > > > want is a full compiler optimization barrier(). > > > > No. See above. > > True, *(volatile foo *)& _will_ work for this case. > > But multiple calls to barrier() (granted, would invalidate all other > optimizations also) would work as well, would it not? They work, but are a bit slower. So they do work, but not as well. > [ Interestingly, if you declared all those objects mentioned earlier as > atomic_t, and x86(-64) switched to an __asm__ __volatile__ based variant > for atomic_{read,set}_volatile(), the bugs you want to avoid would still > be there. "volatile" the C language type-qualifier does have compiler > re-ordering semantics you mentioned earlier, but the "volatile" that > applies to inline asm()s gives no re-ordering guarantees. ] Well, that certainly would be a point in favor of "volatile" over inline asms. ;-) > > > > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting + 1" > > > > was ordered by the compiler to follow the > > > > "ORDERED_WRT_IRQ(me->rcu_flipctr_idx) = idx", and an NMI/SMI > > > > happened between the two, then an __rcu_read_lock() in the NMI/SMI > > > > would incorrectly take the "else" clause of the enclosing "if" > > > > statement. If some other CPU flipped the rcu_ctrlblk.completed > > > > in the meantime, then the __rcu_read_lock() would (correctly) > > > > write the new value into rcu_flipctr_idx. > > > > > > > > Well and good so far. But the problem arises in > > > > __rcu_read_unlock(), which then decrements the wrong counter. > > > > Depending on exactly how subsequent events played out, this could > > > > result in either prematurely ending grace periods or never-ending > > > > grace periods, both of which are fatal outcomes. > > > > > > > > And the following are not needed in the current version of the > > > > patch, but will be in a future version that either avoids disabling > > > > irqs or that dispenses with the smp_read_barrier_depends() that I > > > > have 99% convinced myself is unneeded: > > > > > > > > o nesting = ORDERED_WRT_IRQ(me->rcu_read_lock_nesting); > > > > > > > > o idx = ORDERED_WRT_IRQ(rcu_ctrlblk.completed) & 0x1; > > > > > > > > Furthermore, in that future version, irq handlers can cause the same > > > > mischief that SMI/NMI handlers can in this version. > > So don't remove the local_irq_save/restore, which is well-established and > well-understood for such cases (it doesn't help you with SMI/NMI, > admittedly). This isn't really about RCU or per-cpu vars as such, it's > just about racy code where you don't want to get hit by a concurrent > interrupt (it does turn out that doing things in a _particular order_ will > not cause fatal/buggy behaviour, but it's still a race issue, after all). The local_irq_save/restore are something like 30% of the overhead of these two functions, so will be looking hard at getting rid of them. Doing so allows the scheduling-clock interrupt to get into the mix, and also allows preemption. The goal would be to find some trick that suppresses preemption, fends off the grace-period-computation code invoked from the the scheduling-clock interrupt, and otherwise keeps things on an even keel. > > > > Next, looking at __rcu_read_unlock(): > > > > > > > > o If "ORDERED_WRT_IRQ(me->rcu_read_lock_nesting) = nesting - 1" > > > > was reordered by the compiler to follow the > > > > "ORDERED_WRT_IRQ(__get_cpu_var(rcu_flipctr)[idx])--", > > > > then if an NMI/SMI containing an rcu_read_lock() occurs between > > > > the two, this nested rcu_read_lock() would incorrectly believe > > > > that it was protected by an enclosing RCU read-side critical > > > > section as described in the first reversal discussed for > > > > __rcu_read_lock() above. Again, fatal outcome. > > > > > > > > This is what we have now. It is not hard to imagine situations that > > > > interact with -both- interrupt handlers -and- other CPUs, as described > > > > earlier. > > Unless somebody's going for a lockless implementation, such situations > normally use spin_lock_irqsave() based locking (or local_irq_save for > those who care only for current CPU) -- problem with the patch in question, > is that you want to prevent races with concurrent SMI/NMIs as well, which > is not something that a lot of code needs to consider. Or that needs to resolve similar races with IRQs without disabling them. One reason to avoid disabling IRQs is to avoid degrading scheduling latency. In any case, I do agree that the amount of code that must worry about this is quite small at the moment. I believe that it will become more common, but would imagine that this belief might not be universal. Yet, anyway. ;-) > [ Curiously, another thread is discussing something similar also: > http://lkml.org/lkml/2007/8/15/393 "RFC: do get_rtc_time() correctly" ] > > Anyway, I didn't look at the code in that patch very much in detail, but > why couldn't you implement some kind of synchronization variable that lets > rcu_read_lock() or rcu_read_unlock() -- when being called from inside an > NMI or SMI handler -- know that it has concurrently interrupted an ongoing > rcu_read_{un}lock() and so must do things differently ... (?) Given some low-level details of the current implementation, I could imagine manipulating rcu_read_lock_nesting on entry to and exit from all NMI/SMI handlers, but would like to avoid that kind of architecture dependency. I am not confident of locating all of them, for one thing... > I'm also wondering if there's other code that's not using locking in the > kernel that faces similar issues, and what they've done to deal with it > (if anything). Such bugs would be subtle, and difficult to diagnose. Agreed! Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 16:34 ` Paul E. McKenney 2007-08-16 23:59 ` Herbert Xu @ 2007-08-17 3:15 ` Nick Piggin 2007-08-17 4:02 ` Paul Mackerras ` (2 more replies) 1 sibling, 3 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 3:15 UTC (permalink / raw) To: paulmck Cc: Herbert Xu, Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Paul E. McKenney wrote: > On Thu, Aug 16, 2007 at 06:42:50PM +0800, Herbert Xu wrote: >>In fact, volatile doesn't guarantee that the memory gets >>read anyway. You might be reading some stale value out >>of the cache. Granted this doesn't happen on x86 but >>when you're coding for the kernel you can't make such >>assumptions. >> >>So the point here is that if you don't mind getting a stale >>value from the CPU cache when doing an atomic_read, then >>surely you won't mind getting a stale value from the compiler >>"cache". > > > Absolutely disagree. An interrupt/NMI/SMI handler running on the CPU > will see the same value (whether in cache or in store buffer) that > the mainline code will see. In this case, we don't care about CPU > misordering, only about compiler misordering. It is easy to see > other uses that combine communication with handlers on the current > CPU with communication among CPUs -- again, see prior messages in > this thread. I still don't agree with the underlying assumption that everybody (or lots of kernel code) treats atomic accesses as volatile. Nobody that does has managed to explain my logic problem either: loads and stores to long and ptr have always been considered to be atomic, test_bit is atomic; so why are another special subclass of atomic loads and stores? (and yes, it is perfectly legitimate to want a non-volatile read for a data type that you also want to do atomic RMW operations on) Why are people making these undocumented and just plain false assumptions about atomic_t? If they're using lockless code (ie. which they must be if using atomics), then they actually need to be thinking much harder about memory ordering issues. If that is too much for them, then they can just use locks. >>>So, the architecture guys can implement atomic_read however they want >>>--- as long as it cannot be optimized away.* >> >>They can implement it however they want as long as it stays >>atomic. > > > Precisely. And volatility is a key property of "atomic". Let's please > not throw it away. It isn't, though (at least not since i386 and x86-64 don't have it). _Adding_ it is trivial, and can be done any time. Throwing it away (ie. making the API weaker) is _hard_. So let's not add it without really good reasons. It most definitely results in worse code generation in practice. I don't know why people would assume volatile of atomics. AFAIK, most of the documentation is pretty clear that all the atomic stuff can be reordered etc. except for those that modify and return a value. It isn't even intuitive: `*lp = value` is like the most fundamental atomic operation in Linux. -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:15 ` Nick Piggin @ 2007-08-17 4:02 ` Paul Mackerras 2007-08-17 4:39 ` Nick Piggin 2007-08-17 7:25 ` Stefan Richter 2007-08-17 22:14 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Segher Boessenkool 2 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 4:02 UTC (permalink / raw) To: Nick Piggin Cc: paulmck, Herbert Xu, Stefan Richter, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Nick Piggin writes: > Why are people making these undocumented and just plain false > assumptions about atomic_t? Well, it has only been false since December 2006. Prior to that atomics *were* volatile on all platforms. > If they're using lockless code (ie. > which they must be if using atomics), then they actually need to be > thinking much harder about memory ordering issues. Indeed. I believe that most uses of atomic_read other than in polling loops or debug printk statements are actually racy. In some cases the race doesn't seem to matter, but I'm sure there are cases where it does. > If that is too > much for them, then they can just use locks. Why use locks when you can just sprinkle magic fix-the-races dust (aka atomic_t) over your code? :) :) > > Precisely. And volatility is a key property of "atomic". Let's please > > not throw it away. > > It isn't, though (at least not since i386 and x86-64 don't have it). Conceptually it is, because atomic_t is specifically for variables which are liable to be modified by other CPUs, and volatile _means_ "liable to be changed by mechanisms outside the knowledge of the compiler". > _Adding_ it is trivial, and can be done any time. Throwing it away > (ie. making the API weaker) is _hard_. So let's not add it without Well, in one sense it's not that hard - Linus did it just 8 months ago in commit f9e9dcb3. :) > really good reasons. It most definitely results in worse code > generation in practice. 0.0008% increase in kernel text size on powerpc according to my measurement. :) > I don't know why people would assume volatile of atomics. AFAIK, most By making something an atomic_t you're saying "other CPUs are going to be modifying this, so treat it specially". It's reasonable to assume that special treatment extends to reading and setting it. > of the documentation is pretty clear that all the atomic stuff can be > reordered etc. except for those that modify and return a value. Volatility isn't primarily about reordering (though as Linus says it does restrict reordering to some extent). Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 4:02 ` Paul Mackerras @ 2007-08-17 4:39 ` Nick Piggin 0 siblings, 0 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 4:39 UTC (permalink / raw) To: Paul Mackerras Cc: paulmck, Herbert Xu, Stefan Richter, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Paul Mackerras wrote: > Nick Piggin writes: > > >>Why are people making these undocumented and just plain false >>assumptions about atomic_t? > > > Well, it has only been false since December 2006. Prior to that > atomics *were* volatile on all platforms. Hmm, although I don't think it has ever been guaranteed by the API documentation (concede documentation is often not treated as the authoritative source here, but for atomic it is actually very good and obviously indispensable as the memory ordering reference). >>If they're using lockless code (ie. >>which they must be if using atomics), then they actually need to be >>thinking much harder about memory ordering issues. > > > Indeed. I believe that most uses of atomic_read other than in polling > loops or debug printk statements are actually racy. In some cases the > race doesn't seem to matter, but I'm sure there are cases where it > does. > > >>If that is too >>much for them, then they can just use locks. > > > Why use locks when you can just sprinkle magic fix-the-races dust (aka > atomic_t) over your code? :) :) I agree with your skepticism of a lot of lockless code. But I think a lot of the more subtle race problems will not be fixed with volatile. The big, dumb infinite loop bugs would be fixed, but they're pretty trivial to debug and even audit for. >>>Precisely. And volatility is a key property of "atomic". Let's please >>>not throw it away. >> >>It isn't, though (at least not since i386 and x86-64 don't have it). > > > Conceptually it is, because atomic_t is specifically for variables > which are liable to be modified by other CPUs, and volatile _means_ > "liable to be changed by mechanisms outside the knowledge of the > compiler". Usually that is the case, yes. But also most of the time we don't care that it has been changed and don't mind it being reordered or eliminated. One of the only places we really care about that at all is for variables that are modified by the *same* CPU. >>_Adding_ it is trivial, and can be done any time. Throwing it away >>(ie. making the API weaker) is _hard_. So let's not add it without > > > Well, in one sense it's not that hard - Linus did it just 8 months ago > in commit f9e9dcb3. :) Well it would have been harder if the documentation also guaranteed that atomic_read/atomic_set was ordered. Or it would have been harder for _me_ to make such a change, anyway ;) >>really good reasons. It most definitely results in worse code >>generation in practice. > > > 0.0008% increase in kernel text size on powerpc according to my > measurement. :) I don't think you're making a bad choice by keeping it volatile on powerpc and waiting for others to shake out more of the bugs. You get to fix everybody else's memory ordering bugs :) >>I don't know why people would assume volatile of atomics. AFAIK, most > > > By making something an atomic_t you're saying "other CPUs are going to > be modifying this, so treat it specially". It's reasonable to assume > that special treatment extends to reading and setting it. But I don't actually know what that "special treatment" is. Well actually, I do know that operations will never result in a partial modification being exposed. I also know that the operators that do not modify and return are not guaranteed to have any sort of ordering constraints. -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:15 ` Nick Piggin 2007-08-17 4:02 ` Paul Mackerras @ 2007-08-17 7:25 ` Stefan Richter 2007-08-17 8:06 ` Nick Piggin 2007-08-17 22:14 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Segher Boessenkool 2 siblings, 1 reply; 333+ messages in thread From: Stefan Richter @ 2007-08-17 7:25 UTC (permalink / raw) To: Nick Piggin Cc: paulmck, Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Nick Piggin wrote: > I don't know why people would assume volatile of atomics. AFAIK, most > of the documentation is pretty clear that all the atomic stuff can be > reordered etc. except for those that modify and return a value. Which documentation is there? For driver authors, there is LDD3. It doesn't specifically cover effects of optimization on accesses to atomic_t. For architecture port authors, there is Documentation/atomic_ops.txt. Driver authors also can learn something from that document, as it indirectly documents the atomic_t and bitops APIs. Prompted by this thread, I reread this document, and indeed, the sentence "Unlike the above routines, it is required that explicit memory barriers are performed before and after [atomic_{inc,dec}_return]" indicates that atomic_read (one of the "above routines") is very different from all other atomic_t accessors that return values. This is strange. Why is it that atomic_read stands out that way? IMO this API imbalance is quite unexpected by many people. Wouldn't it be beneficial to change the atomic_read API to behave the same like all other atomic_t accessors that return values? OK, it is also different from the other accessors that return data in so far as it doesn't modify the data. But as driver "author", i.e. user of the API, I can't see much use of an atomic_read that can be reordered and, more importantly, can be optimized away by the compiler. Sure, now that I learned of these properties I can start to audit code and insert barriers where I believe they are needed, but this simply means that almost all occurrences of atomic_read will get barriers (unless there already are implicit but more or less obvious barriers like msleep). -- Stefan Richter -=====-=-=== =--- =---= http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 7:25 ` Stefan Richter @ 2007-08-17 8:06 ` Nick Piggin 2007-08-17 8:58 ` Satyam Sharma ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 8:06 UTC (permalink / raw) To: Stefan Richter Cc: paulmck, Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Stefan Richter wrote: > Nick Piggin wrote: > >>I don't know why people would assume volatile of atomics. AFAIK, most >>of the documentation is pretty clear that all the atomic stuff can be >>reordered etc. except for those that modify and return a value. > > > Which documentation is there? Documentation/atomic_ops.txt > For driver authors, there is LDD3. It doesn't specifically cover > effects of optimization on accesses to atomic_t. > > For architecture port authors, there is Documentation/atomic_ops.txt. > Driver authors also can learn something from that document, as it > indirectly documents the atomic_t and bitops APIs. > "Semantics and Behavior of Atomic and Bitmask Operations" is pretty direct :) Sure, it says that it's for arch maintainers, but there is no reason why users can't make use of it. > Prompted by this thread, I reread this document, and indeed, the > sentence "Unlike the above routines, it is required that explicit memory > barriers are performed before and after [atomic_{inc,dec}_return]" > indicates that atomic_read (one of the "above routines") is very > different from all other atomic_t accessors that return values. > > This is strange. Why is it that atomic_read stands out that way? IMO It is not just atomic_read of course. It is atomic_add,sub,inc,dec,set. > this API imbalance is quite unexpected by many people. Wouldn't it be > beneficial to change the atomic_read API to behave the same like all > other atomic_t accessors that return values? It is very consistent and well defined. Operations which both modify the data _and_ return something are defined to have full barriers before and after. What do you want to add to the other atomic accessors? Full memory barriers? Only compiler barriers? It's quite likely that if you think some barriers will fix bugs, then there are other bugs lurking there anyway. Just use spinlocks if you're not absolutely clear about potential races and memory ordering issues -- they're pretty cheap and simple. > OK, it is also different from the other accessors that return data in so > far as it doesn't modify the data. But as driver "author", i.e. user of > the API, I can't see much use of an atomic_read that can be reordered > and, more importantly, can be optimized away by the compiler. It will return to you an atomic snapshot of the data (loaded from memory at some point since the last compiler barrier). All you have to be aware of compiler barriers and the Linux SMP memory ordering model, which should be a given if you are writing lockless code. > Sure, now > that I learned of these properties I can start to audit code and insert > barriers where I believe they are needed, but this simply means that > almost all occurrences of atomic_read will get barriers (unless there > already are implicit but more or less obvious barriers like msleep). You might find that these places that appear to need barriers are buggy for other reasons anyway. Can you point to some in-tree code we can have a look at? -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:06 ` Nick Piggin @ 2007-08-17 8:58 ` Satyam Sharma 2007-08-17 9:15 ` Nick Piggin 2007-08-17 10:48 ` Stefan Richter 2007-08-18 14:35 ` LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures) Stefan Richter 2 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 8:58 UTC (permalink / raw) To: Nick Piggin Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > Stefan Richter wrote: > [...] > Just use spinlocks if you're not absolutely clear about potential > races and memory ordering issues -- they're pretty cheap and simple. I fully agree with this. As Paul Mackerras mentioned elsewhere, a lot of authors sprinkle atomic_t in code thinking they're somehow done with *locking*. This is sad, and I wonder if it's time for a Documentation/atomic-considered-dodgy.txt kind of document :-) > > Sure, now > > that I learned of these properties I can start to audit code and insert > > barriers where I believe they are needed, but this simply means that > > almost all occurrences of atomic_read will get barriers (unless there > > already are implicit but more or less obvious barriers like msleep). > > You might find that these places that appear to need barriers are > buggy for other reasons anyway. Can you point to some in-tree code > we can have a look at? Such code was mentioned elsewhere (query nodemgr_host_thread in cscope) that managed to escape the requirement for a barrier only because of some completely un-obvious compilation-unit-scope thing. But I find such an non-explicit barrier quite bad taste. Stefan, do consider plunking an explicit call to barrier() there. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:58 ` Satyam Sharma @ 2007-08-17 9:15 ` Nick Piggin 2007-08-17 10:03 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-17 9:15 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > > On Fri, 17 Aug 2007, Nick Piggin wrote: >>>Sure, now >>>that I learned of these properties I can start to audit code and insert >>>barriers where I believe they are needed, but this simply means that >>>almost all occurrences of atomic_read will get barriers (unless there >>>already are implicit but more or less obvious barriers like msleep). >> >>You might find that these places that appear to need barriers are >>buggy for other reasons anyway. Can you point to some in-tree code >>we can have a look at? > > > Such code was mentioned elsewhere (query nodemgr_host_thread in cscope) > that managed to escape the requirement for a barrier only because of > some completely un-obvious compilation-unit-scope thing. But I find such > an non-explicit barrier quite bad taste. Stefan, do consider plunking an > explicit call to barrier() there. It is very obvious. msleep calls schedule() (ie. sleeps), which is always a barrier. The "unobvious" thing is that you wanted to know how the compiler knows a function is a barrier -- answer is that if it does not *know* it is not a barrier, it must assume it is a barrier. If the whole msleep call chain including the scheduler were defined static in the current compilation unit, then it would still be a barrier because it would actually be able to see the barriers in schedule(void), if nothing else. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 9:15 ` Nick Piggin @ 2007-08-17 10:03 ` Satyam Sharma 2007-08-17 11:50 ` Nick Piggin 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 10:03 UTC (permalink / raw) To: Nick Piggin Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > > > > On Fri, 17 Aug 2007, Nick Piggin wrote: > > > > > Sure, now > > > > that I learned of these properties I can start to audit code and insert > > > > barriers where I believe they are needed, but this simply means that > > > > almost all occurrences of atomic_read will get barriers (unless there > > > > already are implicit but more or less obvious barriers like msleep). > > > > > > You might find that these places that appear to need barriers are > > > buggy for other reasons anyway. Can you point to some in-tree code > > > we can have a look at? > > > > > > Such code was mentioned elsewhere (query nodemgr_host_thread in cscope) > > that managed to escape the requirement for a barrier only because of > > some completely un-obvious compilation-unit-scope thing. But I find such > > an non-explicit barrier quite bad taste. Stefan, do consider plunking an > > explicit call to barrier() there. > > It is very obvious. msleep calls schedule() (ie. sleeps), which is > always a barrier. Probably you didn't mean that, but no, schedule() is not barrier because it sleeps. It's a barrier because it's invisible. > The "unobvious" thing is that you wanted to know how the compiler knows > a function is a barrier -- answer is that if it does not *know* it is not > a barrier, it must assume it is a barrier. True, that's clearly what happens here. But are you're definitely joking that this is "obvious" in terms of code-clarity, right? Just 5 minutes back you mentioned elsewhere you like seeing lots of explicit calls to barrier() (with comments, no less, hmm? :-) ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 10:03 ` Satyam Sharma @ 2007-08-17 11:50 ` Nick Piggin 2007-08-17 12:50 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-17 11:50 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > >On Fri, 17 Aug 2007, Nick Piggin wrote: > > >>Satyam Sharma wrote: >> >>It is very obvious. msleep calls schedule() (ie. sleeps), which is >>always a barrier. >> > >Probably you didn't mean that, but no, schedule() is not barrier because >it sleeps. It's a barrier because it's invisible. > Where did I say it is a barrier because it sleeps? It is always a barrier because, at the lowest level, schedule() (and thus anything that sleeps) is defined to always be a barrier. Regardless of whatever obscure means the compiler might need to infer the barrier. In other words, you can ignore those obscure details because schedule() is always going to have an explicit barrier in it. >>The "unobvious" thing is that you wanted to know how the compiler knows >>a function is a barrier -- answer is that if it does not *know* it is not >>a barrier, it must assume it is a barrier. >> > >True, that's clearly what happens here. But are you're definitely joking >that this is "obvious" in terms of code-clarity, right? > No. If you accept that barrier() is implemented correctly, and you know that sleeping is defined to be a barrier, then its perfectly clear. You don't have to know how the compiler "knows" that some function contains a barrier. >Just 5 minutes back you mentioned elsewhere you like seeing lots of >explicit calls to barrier() (with comments, no less, hmm? :-) > Sure, but there are well known primitives which contain barriers, and trivial recognisable code sequences for which you don't need comments. waiting-loops using sleeps or cpu_relax() are prime examples. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 11:50 ` Nick Piggin @ 2007-08-17 12:50 ` Satyam Sharma 2007-08-17 12:56 ` Nick Piggin 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 12:50 UTC (permalink / raw) To: Nick Piggin Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > > On Fri, 17 Aug 2007, Nick Piggin wrote: > > > Satyam Sharma wrote: > > > > > > It is very obvious. msleep calls schedule() (ie. sleeps), which is > > > always a barrier. > > > > Probably you didn't mean that, but no, schedule() is not barrier because > > it sleeps. It's a barrier because it's invisible. > > Where did I say it is a barrier because it sleeps? Just below. What you wrote: > It is always a barrier because, at the lowest level, schedule() (and thus > anything that sleeps) is defined to always be a barrier. "It is always a barrier because, at the lowest level, anything that sleeps is defined to always be a barrier". > Regardless of > whatever obscure means the compiler might need to infer the barrier. > > In other words, you can ignore those obscure details because schedule() is > always going to have an explicit barrier in it. I didn't quite understand what you said here, so I'll tell what I think: * foo() is a compiler barrier if the definition of foo() is invisible to the compiler at a callsite. * foo() is also a compiler barrier if the definition of foo() includes a barrier, and it is inlined at the callsite. If the above is wrong, or if there's something else at play as well, do let me know. > > > The "unobvious" thing is that you wanted to know how the compiler knows > > > a function is a barrier -- answer is that if it does not *know* it is not > > > a barrier, it must assume it is a barrier. > > > > True, that's clearly what happens here. But are you're definitely joking > > that this is "obvious" in terms of code-clarity, right? > > No. If you accept that barrier() is implemented correctly, and you know > that sleeping is defined to be a barrier, Curiously, that's the second time you've said "sleeping is defined to be a (compiler) barrier". How does the compiler even know if foo() is a function that "sleeps"? Do compilers have some notion of "sleeping" to ensure they automatically assume a compiler barrier whenever such a function is called? Or are you saying that the compiler can see the barrier() inside said function ... nopes, you're saying quite the opposite below. > then its perfectly clear. You > don't have to know how the compiler "knows" that some function contains > a barrier. I think I do, why not? Would appreciate if you could elaborate on this. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 12:50 ` Satyam Sharma @ 2007-08-17 12:56 ` Nick Piggin 0 siblings, 0 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 12:56 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > >On Fri, 17 Aug 2007, Nick Piggin wrote: > > >>Satyam Sharma wrote: >> >>>On Fri, 17 Aug 2007, Nick Piggin wrote: >>> >>>>Satyam Sharma wrote: >>>> >>>>It is very obvious. msleep calls schedule() (ie. sleeps), which is >>>>always a barrier. >>>> >>>Probably you didn't mean that, but no, schedule() is not barrier because >>>it sleeps. It's a barrier because it's invisible. >>> >>Where did I say it is a barrier because it sleeps? >> > >Just below. What you wrote: > > >>It is always a barrier because, at the lowest level, schedule() (and thus >>anything that sleeps) is defined to always be a barrier. >> > >"It is always a barrier because, at the lowest level, anything that sleeps >is defined to always be a barrier". > ... because it must call schedule and schedule is a barrier. >>Regardless of >>whatever obscure means the compiler might need to infer the barrier. >> >>In other words, you can ignore those obscure details because schedule() is >>always going to have an explicit barrier in it. >> > >I didn't quite understand what you said here, so I'll tell what I think: > >* foo() is a compiler barrier if the definition of foo() is invisible to > the compiler at a callsite. > >* foo() is also a compiler barrier if the definition of foo() includes > a barrier, and it is inlined at the callsite. > >If the above is wrong, or if there's something else at play as well, >do let me know. > Right. >>>>The "unobvious" thing is that you wanted to know how the compiler knows >>>>a function is a barrier -- answer is that if it does not *know* it is not >>>>a barrier, it must assume it is a barrier. >>>> >>>True, that's clearly what happens here. But are you're definitely joking >>>that this is "obvious" in terms of code-clarity, right? >>> >>No. If you accept that barrier() is implemented correctly, and you know >>that sleeping is defined to be a barrier, >> > >Curiously, that's the second time you've said "sleeping is defined to >be a (compiler) barrier". > _In Linux,_ sleeping is defined to be a compiler barrier. >How does the compiler even know if foo() is >a function that "sleeps"? Do compilers have some notion of "sleeping" >to ensure they automatically assume a compiler barrier whenever such >a function is called? Or are you saying that the compiler can see the >barrier() inside said function ... nopes, you're saying quite the >opposite below. > You're getting too worried about the compiler implementation. Start by assuming that it does work ;) >>then its perfectly clear. You >>don't have to know how the compiler "knows" that some function contains >>a barrier. >> > >I think I do, why not? Would appreciate if you could elaborate on this. > If a function is not completely visible to the compiler (so it can't determine whether a barrier could be in it or not), then it must always assume it will contain a barrier so it always does the right thing. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-17 12:56 ` Nick Piggin 0 siblings, 0 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 12:56 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > >On Fri, 17 Aug 2007, Nick Piggin wrote: > > >>Satyam Sharma wrote: >> >>>On Fri, 17 Aug 2007, Nick Piggin wrote: >>> >>>>Satyam Sharma wrote: >>>> >>>>It is very obvious. msleep calls schedule() (ie. sleeps), which is >>>>always a barrier. >>>> >>>Probably you didn't mean that, but no, schedule() is not barrier because >>>it sleeps. It's a barrier because it's invisible. >>> >>Where did I say it is a barrier because it sleeps? >> > >Just below. What you wrote: > > >>It is always a barrier because, at the lowest level, schedule() (and thus >>anything that sleeps) is defined to always be a barrier. >> > >"It is always a barrier because, at the lowest level, anything that sleeps >is defined to always be a barrier". > ... because it must call schedule and schedule is a barrier. >>Regardless of >>whatever obscure means the compiler might need to infer the barrier. >> >>In other words, you can ignore those obscure details because schedule() is >>always going to have an explicit barrier in it. >> > >I didn't quite understand what you said here, so I'll tell what I think: > >* foo() is a compiler barrier if the definition of foo() is invisible to > the compiler at a callsite. > >* foo() is also a compiler barrier if the definition of foo() includes > a barrier, and it is inlined at the callsite. > >If the above is wrong, or if there's something else at play as well, >do let me know. > Right. >>>>The "unobvious" thing is that you wanted to know how the compiler knows >>>>a function is a barrier -- answer is that if it does not *know* it is not >>>>a barrier, it must assume it is a barrier. >>>> >>>True, that's clearly what happens here. But are you're definitely joking >>>that this is "obvious" in terms of code-clarity, right? >>> >>No. If you accept that barrier() is implemented correctly, and you know >>that sleeping is defined to be a barrier, >> > >Curiously, that's the second time you've said "sleeping is defined to >be a (compiler) barrier". > _In Linux,_ sleeping is defined to be a compiler barrier. >How does the compiler even know if foo() is >a function that "sleeps"? Do compilers have some notion of "sleeping" >to ensure they automatically assume a compiler barrier whenever such >a function is called? Or are you saying that the compiler can see the >barrier() inside said function ... nopes, you're saying quite the >opposite below. > You're getting too worried about the compiler implementation. Start by assuming that it does work ;) >>then its perfectly clear. You >>don't have to know how the compiler "knows" that some function contains >>a barrier. >> > >I think I do, why not? Would appreciate if you could elaborate on this. > If a function is not completely visible to the compiler (so it can't determine whether a barrier could be in it or not), then it must always assume it will contain a barrier so it always does the right thing. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 12:56 ` Nick Piggin (?) @ 2007-08-18 2:15 ` Satyam Sharma -1 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-18 2:15 UTC (permalink / raw) To: Nick Piggin Cc: Stefan Richter, paulmck, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > > > I didn't quite understand what you said here, so I'll tell what I think: > > > > * foo() is a compiler barrier if the definition of foo() is invisible to > > the compiler at a callsite. > > > > * foo() is also a compiler barrier if the definition of foo() includes > > a barrier, and it is inlined at the callsite. > > > > If the above is wrong, or if there's something else at play as well, > > do let me know. > > [...] > If a function is not completely visible to the compiler (so it can't > determine whether a barrier could be in it or not), then it must always > assume it will contain a barrier so it always does the right thing. Yup, that's what I'd said just a few sentences above, as you can see. I was actually asking for "elaboration" on "how a compiler determines that function foo() (say foo == schedule), even when it cannot see that it has a barrier(), as you'd mentioned, is a 'sleeping' function" actually, and whether compilers have a "notion of sleep to automatically assume a compiler barrier whenever such a sleeping function foo() is called". But I think you've already qualified the discussion to this kernel, so okay, I shouldn't nit-pick anymore. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:06 ` Nick Piggin 2007-08-17 8:58 ` Satyam Sharma @ 2007-08-17 10:48 ` Stefan Richter 2007-08-17 10:58 ` Stefan Richter 2007-08-18 14:35 ` LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures) Stefan Richter 2 siblings, 1 reply; 333+ messages in thread From: Stefan Richter @ 2007-08-17 10:48 UTC (permalink / raw) To: Nick Piggin Cc: paulmck, Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Nick Piggin wrote: > Stefan Richter wrote: >> For architecture port authors, there is Documentation/atomic_ops.txt. >> Driver authors also can learn something from that document, as it >> indirectly documents the atomic_t and bitops APIs. > > "Semantics and Behavior of Atomic and Bitmask Operations" is > pretty direct :) "Indirect", "pretty direct"... It's subjective. (It is not an API documentation; it is an implementation specification.) > Sure, it says that it's for arch maintainers, but there is no > reason why users can't make use of it. > > >> Prompted by this thread, I reread this document, and indeed, the >> sentence "Unlike the above routines, it is required that explicit memory >> barriers are performed before and after [atomic_{inc,dec}_return]" >> indicates that atomic_read (one of the "above routines") is very >> different from all other atomic_t accessors that return values. >> >> This is strange. Why is it that atomic_read stands out that way? IMO > > It is not just atomic_read of course. It is atomic_add,sub,inc,dec,set. Yes, but unlike these, atomic_read returns a value. Without me (the API user) providing extra barriers, that value may become something else whenever someone touches code in the vicinity of the atomic_read. >> this API imbalance is quite unexpected by many people. Wouldn't it be >> beneficial to change the atomic_read API to behave the same like all >> other atomic_t accessors that return values? > > It is very consistent and well defined. Operations which both modify > the data _and_ return something are defined to have full barriers > before and after. You are right, atomic_read is not only different from accessors that don't retunr values, it is also different from all other accessors that return values (because they all also modify the value). There is just no actual API documentation, which contributes to the issue that some people (or at least one: me) learn a little bit late how special atomic_read is. > What do you want to add to the other atomic accessors? Full memory > barriers? Only compiler barriers? It's quite likely that if you think > some barriers will fix bugs, then there are other bugs lurking there > anyway. A lot of different though related issues are discussed in this thread, but I personally am only occupied by one particular thing: What kind of return values do I get from atomic_read. > Just use spinlocks if you're not absolutely clear about potential > races and memory ordering issues -- they're pretty cheap and simple. Probably good advice, like generally if driver guys consider lockless algorithms. >> OK, it is also different from the other accessors that return data in so >> far as it doesn't modify the data. But as driver "author", i.e. user of >> the API, I can't see much use of an atomic_read that can be reordered >> and, more importantly, can be optimized away by the compiler. > > It will return to you an atomic snapshot of the data (loaded from > memory at some point since the last compiler barrier). All you have > to be aware of compiler barriers and the Linux SMP memory ordering > model, which should be a given if you are writing lockless code. OK, that's what I slowly realized during this discussion, and I appreciate the explanations that were given here. >> Sure, now >> that I learned of these properties I can start to audit code and insert >> barriers where I believe they are needed, but this simply means that >> almost all occurrences of atomic_read will get barriers (unless there >> already are implicit but more or less obvious barriers like msleep). > > You might find that these places that appear to need barriers are > buggy for other reasons anyway. Can you point to some in-tree code > we can have a look at? I could, or could not, if I were through with auditing the code. I remembered one case and posted it (nodemgr_host_thread) which was safe because msleep_interruptible provided the necessary barrier there, and this implicit barrier is not in danger to be removed by future patches. -- Stefan Richter -=====-=-=== =--- =---= http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 10:48 ` Stefan Richter @ 2007-08-17 10:58 ` Stefan Richter 0 siblings, 0 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-17 10:58 UTC (permalink / raw) To: Nick Piggin Cc: paulmck, Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher I wrote: > Nick Piggin wrote: >> You might find that these places that appear to need barriers are >> buggy for other reasons anyway. Can you point to some in-tree code >> we can have a look at? > > I could, or could not, if I were through with auditing the code. I > remembered one case and posted it (nodemgr_host_thread) which was safe > because msleep_interruptible provided the necessary barrier there, and > this implicit barrier is not in danger to be removed by future patches. PS, just in case anybody holds his breath for more example code from me, I don't plan to continue with an actual audit of the drivers I maintain. It's an important issue, but my current time budget will restrict me to look at it ad hoc, per case. (Open bugs have higher priority than potential bugs.) -- Stefan Richter -=====-=-=== =--- =---= http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures) 2007-08-17 8:06 ` Nick Piggin 2007-08-17 8:58 ` Satyam Sharma 2007-08-17 10:48 ` Stefan Richter @ 2007-08-18 14:35 ` Stefan Richter 2007-08-20 13:28 ` Chris Snook 2 siblings, 1 reply; 333+ messages in thread From: Stefan Richter @ 2007-08-18 14:35 UTC (permalink / raw) To: Jonathan Corbet, Greg Kroah-Hartman Cc: Nick Piggin, paulmck, Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Nick Piggin wrote: > Stefan Richter wrote: >> Nick Piggin wrote: >> >>> I don't know why people would assume volatile of atomics. AFAIK, most >>> of the documentation is pretty clear that all the atomic stuff can be >>> reordered etc. except for those that modify and return a value. >> >> >> Which documentation is there? > > Documentation/atomic_ops.txt > > >> For driver authors, there is LDD3. It doesn't specifically cover >> effects of optimization on accesses to atomic_t. >> >> For architecture port authors, there is Documentation/atomic_ops.txt. >> Driver authors also can learn something from that document, as it >> indirectly documents the atomic_t and bitops APIs. >> > > "Semantics and Behavior of Atomic and Bitmask Operations" is > pretty direct :) > > Sure, it says that it's for arch maintainers, but there is no > reason why users can't make use of it. Note, LDD3 page 238 says: "It is worth noting that most of the other kernel primitives dealing with synchronization, such as spinlock and atomic_t operations, also function as memory barriers." I don't know about Linux 2.6.10 against which LDD3 was written, but currently only _some_ atomic_t operations function as memory barriers. Besides, judging from some posts in this thread, saying that atomic_t operations dealt with synchronization may not be entirely precise. -- Stefan Richter -=====-=-=== =--- =--=- http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures) 2007-08-18 14:35 ` LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures) Stefan Richter @ 2007-08-20 13:28 ` Chris Snook 0 siblings, 0 replies; 333+ messages in thread From: Chris Snook @ 2007-08-20 13:28 UTC (permalink / raw) To: Stefan Richter Cc: Jonathan Corbet, Greg Kroah-Hartman, Nick Piggin, paulmck, Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Stefan Richter wrote: > Nick Piggin wrote: >> Stefan Richter wrote: >>> Nick Piggin wrote: >>> >>>> I don't know why people would assume volatile of atomics. AFAIK, most >>>> of the documentation is pretty clear that all the atomic stuff can be >>>> reordered etc. except for those that modify and return a value. >>> >>> Which documentation is there? >> Documentation/atomic_ops.txt >> >> >>> For driver authors, there is LDD3. It doesn't specifically cover >>> effects of optimization on accesses to atomic_t. >>> >>> For architecture port authors, there is Documentation/atomic_ops.txt. >>> Driver authors also can learn something from that document, as it >>> indirectly documents the atomic_t and bitops APIs. >>> >> "Semantics and Behavior of Atomic and Bitmask Operations" is >> pretty direct :) >> >> Sure, it says that it's for arch maintainers, but there is no >> reason why users can't make use of it. > > > Note, LDD3 page 238 says: "It is worth noting that most of the other > kernel primitives dealing with synchronization, such as spinlock and > atomic_t operations, also function as memory barriers." > > I don't know about Linux 2.6.10 against which LDD3 was written, but > currently only _some_ atomic_t operations function as memory barriers. > > Besides, judging from some posts in this thread, saying that atomic_t > operations dealt with synchronization may not be entirely precise. atomic_t is often used as the basis for implementing more sophisticated synchronization mechanisms, such as rwlocks. Whether or not they are designed for that purpose, the atomic_* operations are de facto synchronization primitives. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:15 ` Nick Piggin 2007-08-17 4:02 ` Paul Mackerras 2007-08-17 7:25 ` Stefan Richter @ 2007-08-17 22:14 ` Segher Boessenkool 2 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 22:14 UTC (permalink / raw) To: Nick Piggin Cc: paulmck, Christoph Lameter, Paul Mackerras, heiko.carstens, Stefan Richter, horms, Satyam Sharma, Linux Kernel Mailing List, rpjday, netdev, ak, cfriesen, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang > (and yes, it is perfectly legitimate to > want a non-volatile read for a data type that you also want to do > atomic RMW operations on) ...which is undefined behaviour in C (and GCC) when that data is declared volatile, which is a good argument against implementing atomics that way in itself. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 10:42 ` Herbert Xu 2007-08-16 16:34 ` Paul E. McKenney @ 2007-08-17 5:04 ` Paul Mackerras 1 sibling, 0 replies; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 5:04 UTC (permalink / raw) To: Herbert Xu Cc: Stefan Richter, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > So the point here is that if you don't mind getting a stale > value from the CPU cache when doing an atomic_read, then > surely you won't mind getting a stale value from the compiler > "cache". No, that particular argument is bogus, because there is a cache coherency protocol operating to keep the CPU cache coherent with stores from other CPUs, but there isn't any such protocol (nor should there be) for a register used as a "cache". (Linux requires SMP systems to keep any CPU caches coherent as far as accesses by other CPUs are concerned. It doesn't support any SMP systems that are not cache-coherent as far as CPU accesses are concerned. It does support systems with non-cache-coherent DMA.) Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 9:54 ` Stefan Richter 2007-08-16 10:31 ` Stefan Richter @ 2007-08-16 10:35 ` Herbert Xu 1 sibling, 0 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-16 10:35 UTC (permalink / raw) To: Stefan Richter Cc: Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 11:54:44AM +0200, Stefan Richter wrote: > > One example was discussed here earlier: The for (;;) loop in > nodemgr_host_thread. There an msleep_interruptible implicitly acted as > barrier (at the moment because it's in a different translation unit; if > it were the same, then because it hopefully has own barriers). So that > happens to work, although such an implicit barrier is bad style: Better > enforce the desired behaviour (== guaranteed load operation) *explicitly*. Hmm, it's not bad style at all. Let's assume that everything is in the same scope. Such a loop must either call a function that busy-waits, which should always have a cpu_relax or something equivalent, or it'll call a function that schedules away which immediately invalidates any values the compiler might have cached for the atomic_read. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 8:10 ` Herbert Xu 2007-08-16 9:54 ` Stefan Richter @ 2007-08-16 19:48 ` Chris Snook 2007-08-17 0:02 ` Herbert Xu 2007-08-17 5:09 ` Paul Mackerras 2 siblings, 1 reply; 333+ messages in thread From: Chris Snook @ 2007-08-16 19:48 UTC (permalink / raw) To: Herbert Xu Cc: Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > On Thu, Aug 16, 2007 at 10:06:31AM +0200, Stefan Richter wrote: >>> Do you (or anyone else for that matter) have an example of this? >> The only code I somewhat know, the ieee1394 subsystem, was perhaps >> authored and is currently maintained with the expectation that each >> occurrence of atomic_read actually results in a load operation, i.e. is >> not optimized away. This means all atomic_t (bus generation, packet and >> buffer refcounts, and some other state variables)* and likewise all >> atomic bitops in that subsystem. > > Can you find an actual atomic_read code snippet there that is > broken without the volatile modifier? A whole bunch of atomic_read uses will be broken without the volatile modifier once we start removing barriers that aren't needed if volatile behavior is guaranteed. barrier() clobbers all your registers. volatile atomic_read() only clobbers one register, and more often than not it's a register you wanted to clobber anyway. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 19:48 ` Chris Snook @ 2007-08-17 0:02 ` Herbert Xu 2007-08-17 2:04 ` Chris Snook 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-17 0:02 UTC (permalink / raw) To: Chris Snook Cc: Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 03:48:54PM -0400, Chris Snook wrote: > > >Can you find an actual atomic_read code snippet there that is > >broken without the volatile modifier? > > A whole bunch of atomic_read uses will be broken without the volatile > modifier once we start removing barriers that aren't needed if volatile > behavior is guaranteed. Could you please cite the file/function names so we can see whether removing the barrier makes sense? Thanks, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 0:02 ` Herbert Xu @ 2007-08-17 2:04 ` Chris Snook 2007-08-17 2:13 ` Herbert Xu 2007-08-17 2:31 ` Nick Piggin 0 siblings, 2 replies; 333+ messages in thread From: Chris Snook @ 2007-08-17 2:04 UTC (permalink / raw) To: Herbert Xu Cc: Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > On Thu, Aug 16, 2007 at 03:48:54PM -0400, Chris Snook wrote: >>> Can you find an actual atomic_read code snippet there that is >>> broken without the volatile modifier? >> A whole bunch of atomic_read uses will be broken without the volatile >> modifier once we start removing barriers that aren't needed if volatile >> behavior is guaranteed. > > Could you please cite the file/function names so we can > see whether removing the barrier makes sense? > > Thanks, At a glance, several architectures' implementations of smp_call_function() have one or more legitimate atomic_read() busy-waits that shouldn't be using CPU-relax. Some of them do work in the loop. I'm sure there are plenty more examples that various maintainers could find in their own code. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 2:04 ` Chris Snook @ 2007-08-17 2:13 ` Herbert Xu 2007-08-17 2:31 ` Nick Piggin 1 sibling, 0 replies; 333+ messages in thread From: Herbert Xu @ 2007-08-17 2:13 UTC (permalink / raw) To: Chris Snook Cc: Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 10:04:24PM -0400, Chris Snook wrote: > > >Could you please cite the file/function names so we can > >see whether removing the barrier makes sense? > > At a glance, several architectures' implementations of smp_call_function() > have one or more legitimate atomic_read() busy-waits that shouldn't be > using CPU-relax. Some of them do work in the loop. Care to name one so we can discuss it? Thanks, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 2:04 ` Chris Snook 2007-08-17 2:13 ` Herbert Xu @ 2007-08-17 2:31 ` Nick Piggin 1 sibling, 0 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 2:31 UTC (permalink / raw) To: Chris Snook Cc: Herbert Xu, Stefan Richter, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Chris Snook wrote: > Herbert Xu wrote: > >> On Thu, Aug 16, 2007 at 03:48:54PM -0400, Chris Snook wrote: >> >>>> Can you find an actual atomic_read code snippet there that is >>>> broken without the volatile modifier? >>> >>> A whole bunch of atomic_read uses will be broken without the volatile >>> modifier once we start removing barriers that aren't needed if >>> volatile behavior is guaranteed. >> >> >> Could you please cite the file/function names so we can >> see whether removing the barrier makes sense? >> >> Thanks, > > > At a glance, several architectures' implementations of > smp_call_function() have one or more legitimate atomic_read() busy-waits > that shouldn't be using CPU-relax. Some of them do work in the loop. sh looks like the only one there that would be broken (and that's only because they don't have a cpu_relax there, but it should be added anyway). Sure, if we removed volatile from other architectures, it would be wise to audit arch code because arch maintainers do sometimes make assumptions about their implementation details... however we can assume most generic code is safe without volatile. -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 8:10 ` Herbert Xu 2007-08-16 9:54 ` Stefan Richter 2007-08-16 19:48 ` Chris Snook @ 2007-08-17 5:09 ` Paul Mackerras 2007-08-17 5:32 ` Herbert Xu 2 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 5:09 UTC (permalink / raw) To: Herbert Xu Cc: Stefan Richter, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > Can you find an actual atomic_read code snippet there that is > broken without the volatile modifier? There are some in arch-specific code, for example line 1073 of arch/mips/kernel/smtc.c. On mips, cpu_relax() is just barrier(), so the empty loop body is ok provided that atomic_read actually does the load each time around the loop. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 5:09 ` Paul Mackerras @ 2007-08-17 5:32 ` Herbert Xu 2007-08-17 5:41 ` Paul Mackerras 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-17 5:32 UTC (permalink / raw) To: Paul Mackerras Cc: Stefan Richter, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, Aug 17, 2007 at 03:09:57PM +1000, Paul Mackerras wrote: > Herbert Xu writes: > > > Can you find an actual atomic_read code snippet there that is > > broken without the volatile modifier? > > There are some in arch-specific code, for example line 1073 of > arch/mips/kernel/smtc.c. On mips, cpu_relax() is just barrier(), so > the empty loop body is ok provided that atomic_read actually does the > load each time around the loop. A barrier() is all you need to force the compiler to reread the value. The people advocating volatile in this thread are talking about code that doesn't use barrier()/cpu_relax(). Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 5:32 ` Herbert Xu @ 2007-08-17 5:41 ` Paul Mackerras 2007-08-17 8:28 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 5:41 UTC (permalink / raw) To: Herbert Xu Cc: Stefan Richter, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu writes: > On Fri, Aug 17, 2007 at 03:09:57PM +1000, Paul Mackerras wrote: > > Herbert Xu writes: > > > > > Can you find an actual atomic_read code snippet there that is > > > broken without the volatile modifier? > > > > There are some in arch-specific code, for example line 1073 of > > arch/mips/kernel/smtc.c. On mips, cpu_relax() is just barrier(), so > > the empty loop body is ok provided that atomic_read actually does the > > load each time around the loop. > > A barrier() is all you need to force the compiler to reread > the value. > > The people advocating volatile in this thread are talking > about code that doesn't use barrier()/cpu_relax(). Did you look at it? Here it is: /* Someone else is initializing in parallel - let 'em finish */ while (atomic_read(&idle_hook_initialized) < 1000) ; Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 5:41 ` Paul Mackerras @ 2007-08-17 8:28 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 8:28 UTC (permalink / raw) To: Paul Mackerras Cc: Herbert Xu, Stefan Richter, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Paul Mackerras wrote: > Herbert Xu writes: > > > On Fri, Aug 17, 2007 at 03:09:57PM +1000, Paul Mackerras wrote: > > > Herbert Xu writes: > > > > > > > Can you find an actual atomic_read code snippet there that is > > > > broken without the volatile modifier? > > > > > > There are some in arch-specific code, for example line 1073 of > > > arch/mips/kernel/smtc.c. On mips, cpu_relax() is just barrier(), so > > > the empty loop body is ok provided that atomic_read actually does the > > > load each time around the loop. > > > > A barrier() is all you need to force the compiler to reread > > the value. > > > > The people advocating volatile in this thread are talking > > about code that doesn't use barrier()/cpu_relax(). > > Did you look at it? Here it is: > > /* Someone else is initializing in parallel - let 'em finish */ > while (atomic_read(&idle_hook_initialized) < 1000) > ; Honestly, this thread is suffering from HUGE communication gaps. What Herbert (obviously) meant there was that "this loop could've been okay _without_ using volatile-semantics-atomic_read() also, if only it used cpu_relax()". That does work, because cpu_relax() is _at least_ barrier() on all archs (on some it also emits some arch-dependent "pause" kind of instruction). Now, saying that "MIPS does not have such an instruction so I won't use cpu_relax() for arch-dependent-busy-while-loops in arch/mips/" sounds like a wrong argument, because: tomorrow, such arch's _may_ introduce such an instruction, so naturally, at that time we'd change cpu_relax() appropriately (in reality, we would actually *re-define* cpu_relax() and ensure that the correct version gets pulled in depending on whether the callsite code is legacy or only for the newer such CPUs of said arch, whatever), but loops such as this would remain un-changed, because they never used cpu_relax()! OTOH an argument that said the following would've made a stronger case: "I don't want to use cpu_relax() because that's a full memory clobber barrier() and I have loop-invariants / other variables around in that code that I *don't* want the compiler to forget just because it used cpu_relax(), and hence I will not use cpu_relax() but instead make my atomic_read() itself have "volatility" semantics. Not just that, but I will introduce a cpu_relax_no_barrier() on MIPS, that would be a no-op #define for now, but which may not be so forever, and continue to use that in such busy loops." In general, please read the thread-summary I've tried to do at: http://lkml.org/lkml/2007/8/17/25 Feel free to continue / comment / correct stuff from there, there's too much confusion and circular-arguments happening on this thread otherwise. [ I might've made an incorrect statement there about "volatile" w.r.t. cache on non-x86 archs, I think. ] Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 7:09 ` Herbert Xu 2007-08-16 8:06 ` Stefan Richter @ 2007-08-16 14:48 ` Ilpo Järvinen 2007-08-16 16:19 ` Stefan Richter ` (2 more replies) 1 sibling, 3 replies; 333+ messages in thread From: Ilpo Järvinen @ 2007-08-16 14:48 UTC (permalink / raw) To: Herbert Xu Cc: Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Herbert Xu wrote: > We've been through that already. If it's a busy-wait it > should use cpu_relax. I looked around a bit by using some command lines and ended up wondering if these are equal to busy-wait case (and should be fixed) or not: ./drivers/telephony/ixj.c 6674: while (atomic_read(&j->DSPWrite) > 0) 6675- atomic_dec(&j->DSPWrite); ...besides that, there are couple of more similar cases in the same file (with braces)... -- i. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 14:48 ` Ilpo Järvinen @ 2007-08-16 16:19 ` Stefan Richter 2007-08-16 19:55 ` Chris Snook 2007-08-16 19:55 ` Chris Snook 2 siblings, 0 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-16 16:19 UTC (permalink / raw) To: Ilpo Järvinen Cc: Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Ilpo Järvinen wrote: > I looked around a bit by using some command lines and ended up wondering > if these are equal to busy-wait case (and should be fixed) or not: > > ./drivers/telephony/ixj.c > 6674: while (atomic_read(&j->DSPWrite) > 0) > 6675- atomic_dec(&j->DSPWrite); > > ...besides that, there are couple of more similar cases in the same file > (with braces)... Generally, ixj.c has several occurrences of couples of atomic write and atomic read which potentially do not do what the author wanted. -- Stefan Richter -=====-=-=== =--- =---- http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 14:48 ` Ilpo Järvinen 2007-08-16 16:19 ` Stefan Richter @ 2007-08-16 19:55 ` Chris Snook 2007-08-16 20:20 ` Christoph Lameter 2007-08-16 21:08 ` Luck, Tony 2007-08-16 19:55 ` Chris Snook 2 siblings, 2 replies; 333+ messages in thread From: Chris Snook @ 2007-08-16 19:55 UTC (permalink / raw) To: Ilpo Järvinen Cc: Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Ilpo Järvinen wrote: > On Thu, 16 Aug 2007, Herbert Xu wrote: > >> We've been through that already. If it's a busy-wait it >> should use cpu_relax. > > I looked around a bit by using some command lines and ended up wondering > if these are equal to busy-wait case (and should be fixed) or not: > > ./drivers/telephony/ixj.c > 6674: while (atomic_read(&j->DSPWrite) > 0) > 6675- atomic_dec(&j->DSPWrite); > > ...besides that, there are couple of more similar cases in the same file > (with braces)... atomic_dec() already has volatile behavior everywhere, so this is semantically okay, but this code (and any like it) should be calling cpu_relax() each iteration through the loop, unless there's a compelling reason not to. I'll allow that for some hardware drivers (possibly this one) such a compelling reason may exist, but hardware-independent core subsystems probably have no excuse. If the maintainer of this code doesn't see a compelling reason to add cpu_relax() in this loop, then it should be patched. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 19:55 ` Chris Snook @ 2007-08-16 20:20 ` Christoph Lameter 2007-08-17 1:02 ` Paul E. McKenney ` (2 more replies) 2007-08-16 21:08 ` Luck, Tony 1 sibling, 3 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 20:20 UTC (permalink / raw) To: Chris Snook Cc: Ilpo Järvinen, Herbert Xu, Paul Mackerras, Satyam Sharma, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Chris Snook wrote: > atomic_dec() already has volatile behavior everywhere, so this is semantically > okay, but this code (and any like it) should be calling cpu_relax() each > iteration through the loop, unless there's a compelling reason not to. I'll > allow that for some hardware drivers (possibly this one) such a compelling > reason may exist, but hardware-independent core subsystems probably have no > excuse. No it does not have any volatile semantics. atomic_dec() can be reordered at will by the compiler within the current basic unit if you do not add a barrier. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 20:20 ` Christoph Lameter @ 2007-08-17 1:02 ` Paul E. McKenney 2007-08-17 1:28 ` Herbert Xu 2007-08-17 2:16 ` Paul Mackerras 2007-08-17 17:41 ` Segher Boessenkool 2 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 1:02 UTC (permalink / raw) To: Christoph Lameter Cc: Chris Snook, Ilpo Järvinen, Herbert Xu, Paul Mackerras, Satyam Sharma, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 01:20:26PM -0700, Christoph Lameter wrote: > On Thu, 16 Aug 2007, Chris Snook wrote: > > > atomic_dec() already has volatile behavior everywhere, so this is semantically > > okay, but this code (and any like it) should be calling cpu_relax() each > > iteration through the loop, unless there's a compelling reason not to. I'll > > allow that for some hardware drivers (possibly this one) such a compelling > > reason may exist, but hardware-independent core subsystems probably have no > > excuse. > > No it does not have any volatile semantics. atomic_dec() can be reordered > at will by the compiler within the current basic unit if you do not add a > barrier. Yep. Or you can use atomic_dec_return() instead of using a barrier. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 1:02 ` Paul E. McKenney @ 2007-08-17 1:28 ` Herbert Xu 2007-08-17 5:07 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-17 1:28 UTC (permalink / raw) To: Paul E. McKenney Cc: Christoph Lameter, Chris Snook, Ilpo Järvinen, Paul Mackerras, Satyam Sharma, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 06:02:32PM -0700, Paul E. McKenney wrote: > > Yep. Or you can use atomic_dec_return() instead of using a barrier. Or you could use smp_mb__{before,after}_atomic_dec. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 1:28 ` Herbert Xu @ 2007-08-17 5:07 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 5:07 UTC (permalink / raw) To: Herbert Xu Cc: Christoph Lameter, Chris Snook, Ilpo Järvinen, Paul Mackerras, Satyam Sharma, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, Aug 17, 2007 at 09:28:00AM +0800, Herbert Xu wrote: > On Thu, Aug 16, 2007 at 06:02:32PM -0700, Paul E. McKenney wrote: > > > > Yep. Or you can use atomic_dec_return() instead of using a barrier. > > Or you could use smp_mb__{before,after}_atomic_dec. Yep. That would be an example of a barrier, either in the atomic_dec() itself or in the smp_mb...(). Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 20:20 ` Christoph Lameter 2007-08-17 1:02 ` Paul E. McKenney @ 2007-08-17 2:16 ` Paul Mackerras 2007-08-17 3:03 ` Linus Torvalds 2007-08-17 17:41 ` Segher Boessenkool 2 siblings, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 2:16 UTC (permalink / raw) To: Christoph Lameter Cc: Chris Snook, Ilpo Järvinen, Herbert Xu, Satyam Sharma, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Christoph Lameter writes: > No it does not have any volatile semantics. atomic_dec() can be reordered > at will by the compiler within the current basic unit if you do not add a > barrier. Volatile doesn't mean it can't be reordered; volatile means the accesses can't be eliminated. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 2:16 ` Paul Mackerras @ 2007-08-17 3:03 ` Linus Torvalds 2007-08-17 3:43 ` Paul Mackerras 2007-08-17 22:09 ` Segher Boessenkool 0 siblings, 2 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-17 3:03 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, Chris Snook, Ilpo J?rvinen, Herbert Xu, Satyam Sharma, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Paul Mackerras wrote: > > Volatile doesn't mean it can't be reordered; volatile means the > accesses can't be eliminated. It also does limit re-ordering. Of course, since *normal* accesses aren't necessarily limited wrt re-ordering, the question then becomes one of "with regard to *what* does it limit re-ordering?". A C compiler that re-orders two different volatile accesses that have a sequence point in between them is pretty clearly a buggy compiler. So at a minimum, it limits re-ordering wrt other volatiles (assuming sequence points exists). It also means that the compiler cannot move it speculatively across conditionals, but other than that it's starting to get fuzzy. In general, I'd *much* rather we used barriers. Anything that "depends" on volatile is pretty much set up to be buggy. But I'm certainly also willing to have that volatile inside "atomic_read/atomic_set()" if it avoids code that would otherwise break - ie if it hides a bug. Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:03 ` Linus Torvalds @ 2007-08-17 3:43 ` Paul Mackerras 2007-08-17 3:53 ` Herbert Xu 2007-08-17 22:09 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-17 3:43 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Lameter, Chris Snook, Ilpo J?rvinen, Herbert Xu, Satyam Sharma, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Linus Torvalds writes: > In general, I'd *much* rather we used barriers. Anything that "depends" on > volatile is pretty much set up to be buggy. But I'm certainly also willing > to have that volatile inside "atomic_read/atomic_set()" if it avoids code > that would otherwise break - ie if it hides a bug. The cost of doing so seems to me to be well down in the noise - 44 bytes of extra kernel text on a ppc64 G5 config, and I don't believe the extra few cycles for the occasional extra load would be measurable (they should all hit in the L1 dcache). I don't mind if x86[-64] have atomic_read/set be nonvolatile and find all the missing barriers, but for now on powerpc, I think that not having to find those missing barriers is worth the 0.00076% increase in kernel text size. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:43 ` Paul Mackerras @ 2007-08-17 3:53 ` Herbert Xu 2007-08-17 6:26 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-17 3:53 UTC (permalink / raw) To: Paul Mackerras Cc: Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo J?rvinen, Satyam Sharma, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, Aug 17, 2007 at 01:43:27PM +1000, Paul Mackerras wrote: > > The cost of doing so seems to me to be well down in the noise - 44 > bytes of extra kernel text on a ppc64 G5 config, and I don't believe > the extra few cycles for the occasional extra load would be measurable > (they should all hit in the L1 dcache). I don't mind if x86[-64] have > atomic_read/set be nonvolatile and find all the missing barriers, but > for now on powerpc, I think that not having to find those missing > barriers is worth the 0.00076% increase in kernel text size. BTW, the sort of missing barriers that triggered this thread aren't that subtle. It'll result in a simple lock-up if the loop condition holds upon entry. At which point it's fairly straightforward to find the culprit. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:53 ` Herbert Xu @ 2007-08-17 6:26 ` Satyam Sharma 2007-08-17 8:38 ` Nick Piggin 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 6:26 UTC (permalink / raw) To: Herbert Xu Cc: Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Herbert Xu wrote: > On Fri, Aug 17, 2007 at 01:43:27PM +1000, Paul Mackerras wrote: > > > > The cost of doing so seems to me to be well down in the noise - 44 > > bytes of extra kernel text on a ppc64 G5 config, and I don't believe > > the extra few cycles for the occasional extra load would be measurable > > (they should all hit in the L1 dcache). I don't mind if x86[-64] have > > atomic_read/set be nonvolatile and find all the missing barriers, but > > for now on powerpc, I think that not having to find those missing > > barriers is worth the 0.00076% increase in kernel text size. > > BTW, the sort of missing barriers that triggered this thread > aren't that subtle. It'll result in a simple lock-up if the > loop condition holds upon entry. At which point it's fairly > straightforward to find the culprit. Not necessarily. A barrier-less buggy code such as below: atomic_set(&v, 0); ... /* some initial code */ while (atomic_read(&v)) ; ... /* code that MUST NOT be executed unless v becomes non-zero */ (where v->counter is has no volatile access semantics) could be generated by the compiler to simply *elid* or *do away* with the loop itself, thereby making the: "/* code that MUST NOT be executed unless v becomes non-zero */" to be executed even when v is zero! That is subtle indeed, and causes no hard lockups. Granted, the above IS buggy code. But, the stated objective is to avoid heisenbugs. And we have driver / subsystem maintainers such as Stefan coming up and admitting that often a lot of code that's written to use atomic_read() does assume the read will not be elided by the compiler. See, I agree, "volatility" semantics != what we often want. However, if what we want is compiler barrier, for only the object under consideration, "volatility" semantics aren't really "nonsensical" or anything. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 6:26 ` Satyam Sharma @ 2007-08-17 8:38 ` Nick Piggin 2007-08-17 9:14 ` Satyam Sharma 2007-08-17 11:08 ` Stefan Richter 0 siblings, 2 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 8:38 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > > On Fri, 17 Aug 2007, Herbert Xu wrote: > > >>On Fri, Aug 17, 2007 at 01:43:27PM +1000, Paul Mackerras wrote: >> >>BTW, the sort of missing barriers that triggered this thread >>aren't that subtle. It'll result in a simple lock-up if the >>loop condition holds upon entry. At which point it's fairly >>straightforward to find the culprit. > > > Not necessarily. A barrier-less buggy code such as below: > > atomic_set(&v, 0); > > ... /* some initial code */ > > while (atomic_read(&v)) > ; > > ... /* code that MUST NOT be executed unless v becomes non-zero */ > > (where v->counter is has no volatile access semantics) > > could be generated by the compiler to simply *elid* or *do away* with > the loop itself, thereby making the: > > "/* code that MUST NOT be executed unless v becomes non-zero */" > > to be executed even when v is zero! That is subtle indeed, and causes > no hard lockups. Then I presume you mean while (!atomic_read(&v)) ; Which is just the same old infinite loop bug solved with cpu_relax(). These are pretty trivial to audit and fix, and also to debug, I would think. > Granted, the above IS buggy code. But, the stated objective is to avoid > heisenbugs. Anyway, why are you making up code snippets that are buggy in other ways in order to support this assertion being made that lots of kernel code supposedly depends on volatile semantics. Just reference the actual code. > And we have driver / subsystem maintainers such as Stefan > coming up and admitting that often a lot of code that's written to use > atomic_read() does assume the read will not be elided by the compiler. So these are broken on i386 and x86-64? Are they definitely safe on SMP and weakly ordered machines with just a simple compiler barrier there? Because I would not be surprised if there are a lot of developers who don't really know what to assume when it comes to memory ordering issues. This is not a dig at driver writers: we still have memory ordering problems in the VM too (and probably most of the subtle bugs in lockless VM code are memory ordering ones). Let's not make up a false sense of security and hope that sprinkling volatile around will allow people to write bug-free lockless code. If a writer can't be bothered reading API documentation and learning the Linux memory model, they can still be productive writing safely locked code. -- SUSE Labs, Novell Inc. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:38 ` Nick Piggin @ 2007-08-17 9:14 ` Satyam Sharma 2007-08-17 9:31 ` Nick Piggin 2007-08-17 11:08 ` Stefan Richter 1 sibling, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 9:14 UTC (permalink / raw) To: Nick Piggin Cc: Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > [...] > > Granted, the above IS buggy code. But, the stated objective is to avoid > > heisenbugs. ^^^^^^^^^^ > Anyway, why are you making up code snippets that are buggy in other > ways in order to support this assertion being made that lots of kernel > code supposedly depends on volatile semantics. Just reference the > actual code. Because the point is *not* about existing bugs in kernel code. At some point Chris Snook (who started this thread) did write that "If I knew of the existing bugs in the kernel, I would be sending patches for them, not this series" or something to that effect. The point is about *author expecations*. If people do expect atomic_read() (or a variant thereof) to have volatile semantics, why not give them such a variant? And by the way, the point is *also* about the fact that cpu_relax(), as of today, implies a full memory clobber, which is not what a lot of such loops want. (due to stuff mentioned elsewhere, summarized in that summary) > > And we have driver / subsystem maintainers such as Stefan > > coming up and admitting that often a lot of code that's written to use > > atomic_read() does assume the read will not be elided by the compiler. ^^^^^^^^^^^^^ (so it's about compiler barrier expectations only, though I fully agree that those who're using atomic_t as if it were some magic thing that lets them write lockless code are sorrily mistaken.) > So these are broken on i386 and x86-64? Possibly, but the point is not about existing bugs, as mentioned above. Some such bugs have been found nonetheless -- reminds me, can somebody please apply http://www.gossamer-threads.com/lists/linux/kernel/810674 ? Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 9:14 ` Satyam Sharma @ 2007-08-17 9:31 ` Nick Piggin 2007-08-17 10:55 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Nick Piggin @ 2007-08-17 9:31 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > > On Fri, 17 Aug 2007, Nick Piggin wrote: > > >>Satyam Sharma wrote: >>[...] >> >>>Granted, the above IS buggy code. But, the stated objective is to avoid >>>heisenbugs. > > ^^^^^^^^^^ > > >>Anyway, why are you making up code snippets that are buggy in other >>ways in order to support this assertion being made that lots of kernel >>code supposedly depends on volatile semantics. Just reference the >>actual code. > > > Because the point is *not* about existing bugs in kernel code. At some > point Chris Snook (who started this thread) did write that "If I knew > of the existing bugs in the kernel, I would be sending patches for them, > not this series" or something to that effect. > > The point is about *author expecations*. If people do expect atomic_read() > (or a variant thereof) to have volatile semantics, why not give them such > a variant? Because they should be thinking about them in terms of barriers, over which the compiler / CPU is not to reorder accesses or cache memory operations, rather than "special" "volatile" accesses. Linux's whole memory ordering and locking model is completely geared around the former. > And by the way, the point is *also* about the fact that cpu_relax(), as > of today, implies a full memory clobber, which is not what a lot of such > loops want. (due to stuff mentioned elsewhere, summarized in that summary) That's not the point, because as I also mentioned, the logical extention to Linux's barrier API to handle this is the order(x) macro. Again, not special volatile accessors. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 9:31 ` Nick Piggin @ 2007-08-17 10:55 ` Satyam Sharma 2007-08-17 12:39 ` Nick Piggin 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 10:55 UTC (permalink / raw) To: Nick Piggin Cc: Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > > [...] > > The point is about *author expecations*. If people do expect atomic_read() > > (or a variant thereof) to have volatile semantics, why not give them such > > a variant? > > Because they should be thinking about them in terms of barriers, over > which the compiler / CPU is not to reorder accesses or cache memory > operations, rather than "special" "volatile" accesses. This is obviously just a taste thing. Whether to have that forget(x) barrier as something author should explicitly sprinkle appropriately in appropriate places in the code by himself or use a primitive that includes it itself. I'm not saying "taste matters aren't important" (they are), but I'm really skeptical if most folks would find the former tasteful. > > And by the way, the point is *also* about the fact that cpu_relax(), as > > of today, implies a full memory clobber, which is not what a lot of such > > loops want. (due to stuff mentioned elsewhere, summarized in that summary) > > That's not the point, That's definitely the point, why not. This is why "barrier()", being heavy-handed, is not the best option. > because as I also mentioned, the logical extention > to Linux's barrier API to handle this is the order(x) macro. Again, not > special volatile accessors. Sure, that forget(x) macro _is_ proposed to be made part of the generic API. Doesn't explain why not to define/use primitives that has volatility semantics in itself, though (taste matters apart). ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 10:55 ` Satyam Sharma @ 2007-08-17 12:39 ` Nick Piggin 0 siblings, 0 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 12:39 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > >On Fri, 17 Aug 2007, Nick Piggin wrote: > > >>Because they should be thinking about them in terms of barriers, over >>which the compiler / CPU is not to reorder accesses or cache memory >>operations, rather than "special" "volatile" accesses. >> > >This is obviously just a taste thing. Whether to have that forget(x) >barrier as something author should explicitly sprinkle appropriately >in appropriate places in the code by himself or use a primitive that >includes it itself. > That's not obviously just taste to me. Not when the primitive has many (perhaps, the majority) of uses that do not require said barriers. And this is not solely about the code generation (which, as Paul says, is relatively minor even on x86). I prefer people to think explicitly about barriers in their lockless code. >I'm not saying "taste matters aren't important" (they are), but I'm really >skeptical if most folks would find the former tasteful. > So I /do/ have better taste than most folks? Thanks! :-) >>>And by the way, the point is *also* about the fact that cpu_relax(), as >>>of today, implies a full memory clobber, which is not what a lot of such >>>loops want. (due to stuff mentioned elsewhere, summarized in that summary) >>> >>That's not the point, >> > >That's definitely the point, why not. This is why "barrier()", being >heavy-handed, is not the best option. > That is _not_ the point (of why a volatile atomic_read is good) because there has already been an alternative posted that better conforms with Linux barrier API and is much more widely useful and more usable. If you are so worried about barrier() being too heavyweight, then you're off to a poor start by wanting to add a few K of kernel text by making atomic_read volatile. >>because as I also mentioned, the logical extention >>to Linux's barrier API to handle this is the order(x) macro. Again, not >>special volatile accessors. >> > >Sure, that forget(x) macro _is_ proposed to be made part of the generic >API. Doesn't explain why not to define/use primitives that has volatility >semantics in itself, though (taste matters apart). > If you follow the discussion.... You were thinking of a reason why the semantics *should* be changed or added, and I was rebutting your argument that it must be used when a full barrier() is too heavy (ie. by pointing out that order() has superior semantics anyway). Why do I keep repeating the same things? I'll not continue bloating this thread until a new valid point comes up... ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-17 12:39 ` Nick Piggin 0 siblings, 0 replies; 333+ messages in thread From: Nick Piggin @ 2007-08-17 12:39 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma wrote: > >On Fri, 17 Aug 2007, Nick Piggin wrote: > > >>Because they should be thinking about them in terms of barriers, over >>which the compiler / CPU is not to reorder accesses or cache memory >>operations, rather than "special" "volatile" accesses. >> > >This is obviously just a taste thing. Whether to have that forget(x) >barrier as something author should explicitly sprinkle appropriately >in appropriate places in the code by himself or use a primitive that >includes it itself. > That's not obviously just taste to me. Not when the primitive has many (perhaps, the majority) of uses that do not require said barriers. And this is not solely about the code generation (which, as Paul says, is relatively minor even on x86). I prefer people to think explicitly about barriers in their lockless code. >I'm not saying "taste matters aren't important" (they are), but I'm really >skeptical if most folks would find the former tasteful. > So I /do/ have better taste than most folks? Thanks! :-) >>>And by the way, the point is *also* about the fact that cpu_relax(), as >>>of today, implies a full memory clobber, which is not what a lot of such >>>loops want. (due to stuff mentioned elsewhere, summarized in that summary) >>> >>That's not the point, >> > >That's definitely the point, why not. This is why "barrier()", being >heavy-handed, is not the best option. > That is _not_ the point (of why a volatile atomic_read is good) because there has already been an alternative posted that better conforms with Linux barrier API and is much more widely useful and more usable. If you are so worried about barrier() being too heavyweight, then you're off to a poor start by wanting to add a few K of kernel text by making atomic_read volatile. >>because as I also mentioned, the logical extention >>to Linux's barrier API to handle this is the order(x) macro. Again, not >>special volatile accessors. >> > >Sure, that forget(x) macro _is_ proposed to be made part of the generic >API. Doesn't explain why not to define/use primitives that has volatility >semantics in itself, though (taste matters apart). > If you follow the discussion.... You were thinking of a reason why the semantics *should* be changed or added, and I was rebutting your argument that it must be used when a full barrier() is too heavy (ie. by pointing out that order() has superior semantics anyway). Why do I keep repeating the same things? I'll not continue bloating this thread until a new valid point comes up... ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 12:39 ` Nick Piggin (?) @ 2007-08-17 13:36 ` Satyam Sharma -1 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 13:36 UTC (permalink / raw) To: Nick Piggin Cc: Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > Satyam Sharma wrote: > > > On Fri, 17 Aug 2007, Nick Piggin wrote: > > > > > Because they should be thinking about them in terms of barriers, over > > > which the compiler / CPU is not to reorder accesses or cache memory > > > operations, rather than "special" "volatile" accesses. > > > > This is obviously just a taste thing. Whether to have that forget(x) > > barrier as something author should explicitly sprinkle appropriately > > in appropriate places in the code by himself or use a primitive that > > includes it itself. > > That's not obviously just taste to me. Not when the primitive has many > (perhaps, the majority) of uses that do not require said barriers. And > this is not solely about the code generation (which, as Paul says, is > relatively minor even on x86). See, you do *require* people to have to repeat the same things to you! As has been written about enough times already, and if you followed the discussion on this thread, I am *not* proposing that atomic_read()'s semantics be changed to have any extra barriers. What is proposed is a different atomic_read_xxx() variant thereof, that those can use who do want that. Now whether to have a kind of barrier ("volatile", whatever) in the atomic_read_xxx() itself, or whether to make the code writer himself to explicitly write the order(x) appropriately in appropriate places in the code _is_ a matter of taste. > > That's definitely the point, why not. This is why "barrier()", being > > heavy-handed, is not the best option. > > That is _not_ the point [...] Again, you're requiring me to repeat things that were already made evident on this thread (if you follow it). This _is_ the point, because a lot of loops out there (too many of them, I WILL NOT bother citing file_name:line_number) end up having to use a barrier just because they're using a loop-exit-condition that depends on a value returned by atomic_read(). It would be good for them if they used an atomic_read_xxx() primitive that gave these "volatility" semantics without junking compiler optimizations for other memory references. > because there has already been an alternative posted Whether that alternative (explicitly using forget(x), or wrappers thereof, such as the "order_atomic" you proposed) is better than other alternatives (such as atomic_read_xxx() which includes the volatility behaviour in itself) is still open, and precisely what we started discussing just one mail back. (The above was also mostly stuff I had to repeated for you, sadly.) > that better conforms with Linux barrier > API and is much more widely useful and more usable. I don't think so. (Now *this* _is_ the "taste-dependent matter" that I mentioned earlier.) > If you are so worried > about > barrier() being too heavyweight, then you're off to a poor start by wanting to > add a few K of kernel text by making atomic_read volatile. Repeating myself, for the N'th time, NO, I DON'T want to make atomic_read have "volatile" semantics. > > > because as I also mentioned, the logical extention > > > to Linux's barrier API to handle this is the order(x) macro. Again, not > > > special volatile accessors. > > > > Sure, that forget(x) macro _is_ proposed to be made part of the generic > > API. Doesn't explain why not to define/use primitives that has volatility ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > semantics in itself, though (taste matters apart). ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > If you follow the discussion.... You were thinking of a reason why the > semantics *should* be changed or added, and I was rebutting your argument > that it must be used when a full barrier() is too heavy (ie. by pointing > out that order() has superior semantics anyway). Amazing. Either you have reading comprehension problems, or else, please try reading this thread (or at least this sub-thread) again. I don't want _you_ blaming _me_ for having to repeat things to you all over again. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 12:39 ` Nick Piggin (?) (?) @ 2007-08-17 16:48 ` Linus Torvalds 2007-08-17 18:50 ` Chris Friesen ` (2 more replies) -1 siblings, 3 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-17 16:48 UTC (permalink / raw) To: Nick Piggin Cc: Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Nick Piggin wrote: > > That's not obviously just taste to me. Not when the primitive has many > (perhaps, the majority) of uses that do not require said barriers. And > this is not solely about the code generation (which, as Paul says, is > relatively minor even on x86). I prefer people to think explicitly > about barriers in their lockless code. Indeed. I think the important issues are: - "volatile" itself is simply a badly/weakly defined issue. The semantics of it as far as the compiler is concerned are really not very good, and in practice tends to boil down to "I will generate so bad code that nobody can accuse me of optimizing anything away". - "volatile" - regardless of how well or badly defined it is - is purely a compiler thing. It has absolutely no meaning for the CPU itself, so it at no point implies any CPU barriers. As a result, even if the compiler generates crap code and doesn't re-order anything, there's nothing that says what the CPU will do. - in other words, the *only* possible meaning for "volatile" is a purely single-CPU meaning. And if you only have a single CPU involved in the process, the "volatile" is by definition pointless (because even without a volatile, the compiler is required to make the C code appear consistent as far as a single CPU is concerned). So, let's take the example *buggy* code where we use "volatile" to wait for other CPU's: atomic_set(&var, 0); while (!atomic_read(&var)) /* nothing */; which generates an endless loop if we don't have atomic_read() imply volatile. The point here is that it's buggy whether the volatile is there or not! Exactly because the user expects multi-processing behaviour, but "volatile" doesn't actually give any real guarantees about it. Another CPU may have done: external_ptr = kmalloc(..); /* Setup is now complete, inform the waiter */ atomic_inc(&var); but the fact is, since the other CPU isn't serialized in any way, the "while-loop" (even in the presense of "volatile") doesn't actually work right! Whatever the "atomic_read()" was waiting for may not have completed, because we have no barriers! So if "volatile" makes a difference, it is invariably a sign of a bug in serialization (the one exception is for IO - we use "volatile" to avoid having to use inline asm for IO on x86) - and for "random values" like jiffies). So the question should *not* be whether "volatile" actually fixes bugs. It *never* fixes a bug. But what it can do is to hide the obvious ones. In other words, adding a volaile in the above kind of situation of "atomic_read()" will certainly turn an obvious bug into something that works "practically all of the time). So anybody who argues for "volatile" fixing bugs is fundamentally incorrect. It does NO SUCH THING. By arguing that, such people only show that you have no idea what they are talking about. So the only reason to add back "volatile" to the atomic_read() sequence is not to fix bugs, but to _hide_ the bugs better. They're still there, they are just a lot harder to trigger, and tend to be a lot subtler. And hey, sometimes "hiding bugs well enough" is ok. In this case, I'd argue that we've successfully *not* had the volatile there for eight months on x86-64, and that should tell people something. (Does _removing_ the volatile fix bugs? No - callers still need to think about barriers etc, and lots of people don't. So I'm not claiming that removing volatile fixes any bugs either, but I *am* claiming that: - removing volatile makes some bugs easier to see (which is mostly a good thing: they were there before, anyway). - removing volatile generates better code (which is a good thing, even if it's just 0.1%) - removing volatile removes a huge mental *bug* that lots of people seem to have, as shown by this whole thread. Anybody who thinks that "volatile" actually fixes anything has a gaping hole in their head, and we should remove volatile just to make sure that nobody thinks that it means soemthign that it doesn't mean! In other words, this whole discussion has just convinced me that we should *not* add back "volatile" to "atomic_read()" - I was willing to do it for practical and "hide the bugs" reasons, but having seen people argue for it, thinking that it actually fixes something, I'm now convinced that the *last* thing we should do is to encourage that kind of superstitious thinking. "volatile" is like a black cat crossing the road. Sure, it affects *something* (at a minimum: before, the black cat was on one side of the road, afterwards it is on the other side of the road), but it has no bigger and longer-lasting direct affects. People who think "volatile" really matters are just fooling themselves. Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 16:48 ` Linus Torvalds @ 2007-08-17 18:50 ` Chris Friesen 2007-08-20 13:15 ` Chris Snook 2007-09-09 18:02 ` Denys Vlasenko 2 siblings, 0 replies; 333+ messages in thread From: Chris Friesen @ 2007-08-17 18:50 UTC (permalink / raw) To: Linus Torvalds Cc: Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher Linus Torvalds wrote: > - in other words, the *only* possible meaning for "volatile" is a purely > single-CPU meaning. And if you only have a single CPU involved in the > process, the "volatile" is by definition pointless (because even > without a volatile, the compiler is required to make the C code appear > consistent as far as a single CPU is concerned). I assume you mean "except for IO-related code and 'random' values like jiffies" as you mention later on? I assume other values set in interrupt handlers would count as "random" from a volatility perspective? > So anybody who argues for "volatile" fixing bugs is fundamentally > incorrect. It does NO SUCH THING. By arguing that, such people only show > that you have no idea what they are talking about. What about reading values modified in interrupt handlers, as in your "random" case? Or is this a bug where the user of atomic_read() is invalidly expecting a read each time it is called? Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-17 18:50 ` Chris Friesen 0 siblings, 0 replies; 333+ messages in thread From: Chris Friesen @ 2007-08-17 18:50 UTC (permalink / raw) To: Linus Torvalds Cc: Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher Linus Torvalds wrote: > - in other words, the *only* possible meaning for "volatile" is a purely > single-CPU meaning. And if you only have a single CPU involved in the > process, the "volatile" is by definition pointless (because even > without a volatile, the compiler is required to make the C code appear > consistent as far as a single CPU is concerned). I assume you mean "except for IO-related code and 'random' values like jiffies" as you mention later on? I assume other values set in interrupt handlers would count as "random" from a volatility perspective? > So anybody who argues for "volatile" fixing bugs is fundamentally > incorrect. It does NO SUCH THING. By arguing that, such people only show > that you have no idea what they are talking about. What about reading values modified in interrupt handlers, as in your "random" case? Or is this a bug where the user of atomic_read() is invalidly expecting a read each time it is called? Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 18:50 ` Chris Friesen (?) @ 2007-08-17 18:54 ` Arjan van de Ven 2007-08-17 19:49 ` Paul E. McKenney -1 siblings, 1 reply; 333+ messages in thread From: Arjan van de Ven @ 2007-08-17 18:54 UTC (permalink / raw) To: Chris Friesen Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher On Fri, 2007-08-17 at 12:50 -0600, Chris Friesen wrote: > Linus Torvalds wrote: > > > - in other words, the *only* possible meaning for "volatile" is a purely > > single-CPU meaning. And if you only have a single CPU involved in the > > process, the "volatile" is by definition pointless (because even > > without a volatile, the compiler is required to make the C code appear > > consistent as far as a single CPU is concerned). > > I assume you mean "except for IO-related code and 'random' values like > jiffies" as you mention later on? I assume other values set in > interrupt handlers would count as "random" from a volatility perspective? > > > So anybody who argues for "volatile" fixing bugs is fundamentally > > incorrect. It does NO SUCH THING. By arguing that, such people only show > > that you have no idea what they are talking about. > > What about reading values modified in interrupt handlers, as in your > "random" case? Or is this a bug where the user of atomic_read() is > invalidly expecting a read each time it is called? the interrupt handler case is an SMP case since you do not know beforehand what cpu your interrupt handler will run on. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 18:54 ` Arjan van de Ven @ 2007-08-17 19:49 ` Paul E. McKenney 2007-08-17 19:49 ` Arjan van de Ven 0 siblings, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 19:49 UTC (permalink / raw) To: Arjan van de Ven Cc: Chris Friesen, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher On Fri, Aug 17, 2007 at 11:54:33AM -0700, Arjan van de Ven wrote: > > On Fri, 2007-08-17 at 12:50 -0600, Chris Friesen wrote: > > Linus Torvalds wrote: > > > > > - in other words, the *only* possible meaning for "volatile" is a purely > > > single-CPU meaning. And if you only have a single CPU involved in the > > > process, the "volatile" is by definition pointless (because even > > > without a volatile, the compiler is required to make the C code appear > > > consistent as far as a single CPU is concerned). > > > > I assume you mean "except for IO-related code and 'random' values like > > jiffies" as you mention later on? I assume other values set in > > interrupt handlers would count as "random" from a volatility perspective? > > > > > So anybody who argues for "volatile" fixing bugs is fundamentally > > > incorrect. It does NO SUCH THING. By arguing that, such people only show > > > that you have no idea what they are talking about. > > > > What about reading values modified in interrupt handlers, as in your > > "random" case? Or is this a bug where the user of atomic_read() is > > invalidly expecting a read each time it is called? > > the interrupt handler case is an SMP case since you do not know > beforehand what cpu your interrupt handler will run on. With the exception of per-CPU variables, yes. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 19:49 ` Paul E. McKenney @ 2007-08-17 19:49 ` Arjan van de Ven 2007-08-17 20:12 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Arjan van de Ven @ 2007-08-17 19:49 UTC (permalink / raw) To: paulmck Cc: Chris Friesen, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher On Fri, 2007-08-17 at 12:49 -0700, Paul E. McKenney wrote: > > > What about reading values modified in interrupt handlers, as in your > > > "random" case? Or is this a bug where the user of atomic_read() is > > > invalidly expecting a read each time it is called? > > > > the interrupt handler case is an SMP case since you do not know > > beforehand what cpu your interrupt handler will run on. > > With the exception of per-CPU variables, yes. if you're spinning waiting for a per-CPU variable to get changed by an interrupt handler... you have bigger problems than "volatile" ;-) -- if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 19:49 ` Arjan van de Ven @ 2007-08-17 20:12 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-17 20:12 UTC (permalink / raw) To: Arjan van de Ven Cc: Chris Friesen, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher On Fri, Aug 17, 2007 at 12:49:00PM -0700, Arjan van de Ven wrote: > > On Fri, 2007-08-17 at 12:49 -0700, Paul E. McKenney wrote: > > > > What about reading values modified in interrupt handlers, as in your > > > > "random" case? Or is this a bug where the user of atomic_read() is > > > > invalidly expecting a read each time it is called? > > > > > > the interrupt handler case is an SMP case since you do not know > > > beforehand what cpu your interrupt handler will run on. > > > > With the exception of per-CPU variables, yes. > > if you're spinning waiting for a per-CPU variable to get changed by an > interrupt handler... you have bigger problems than "volatile" ;-) That would be true, if you were doing that. But you might instead be simply making sure that the mainline actions were seen in order by the interrupt handler. My current example is the NMI-save rcu_read_lock() implementation for realtime. Not the common case, I will admit, but still real. ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 18:50 ` Chris Friesen (?) (?) @ 2007-08-17 19:08 ` Linus Torvalds -1 siblings, 0 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-17 19:08 UTC (permalink / raw) To: Chris Friesen Cc: Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, zlynx, rpjday, jesper.juhl, segher On Fri, 17 Aug 2007, Chris Friesen wrote: > > I assume you mean "except for IO-related code and 'random' values like > jiffies" as you mention later on? Yes. There *are* valid uses for "volatile", but they have remained the same for the last few years: - "jiffies" - internal per-architecture IO implementations that can do them as normal stores. > I assume other values set in interrupt handlers would count as "random" > from a volatility perspective? I don't really see any valid case. I can imagine that you have your own "jiffy" counter in a driver, but what's the point, really? I'd suggest not using volatile, and using barriers instead. > > > So anybody who argues for "volatile" fixing bugs is fundamentally > > incorrect. It does NO SUCH THING. By arguing that, such people only > > show that you have no idea what they are talking about. > What about reading values modified in interrupt handlers, as in your > "random" case? Or is this a bug where the user of atomic_read() is > invalidly expecting a read each time it is called? Quite frankly, the biggest reason for using "volatile" on jiffies was really historic. So even the "random" case is not really a very strong one. You'll notice that anybody who is actually careful will be using sequence locks for the jiffy accesses, if only because the *full* jiffy count is actually a 64-bit value, and so you cannot get it atomically on a 32-bit architecture even on a single CPU (ie a timer interrupt might happen in between reading the low and the high word, so "volatile" is only used for the low 32 bits). So even for jiffies, we actually have: extern u64 __jiffy_data jiffies_64; extern unsigned long volatile __jiffy_data jiffies; where the *real* jiffies is not volatile: the volatile one is using linker tricks to alias the low 32 bits: - arch/i386/kernel/vmlinux.lds.S: ... jiffies = jiffies_64; ... and the only reason we do all these games is (a) it works and (b) it's legacy. Note how I do *not* say "(c) it's a good idea". Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 16:48 ` Linus Torvalds @ 2007-08-20 13:15 ` Chris Snook 2007-08-20 13:15 ` Chris Snook 2007-09-09 18:02 ` Denys Vlasenko 2 siblings, 0 replies; 333+ messages in thread From: Chris Snook @ 2007-08-20 13:15 UTC (permalink / raw) To: Linus Torvalds Cc: Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Linus Torvalds wrote: > So the only reason to add back "volatile" to the atomic_read() sequence is > not to fix bugs, but to _hide_ the bugs better. They're still there, they > are just a lot harder to trigger, and tend to be a lot subtler. What about barrier removal? With consistent semantics we could optimize a fair amount of code. Whether or not that constitutes "premature" optimization is open to debate, but there's no question we could reduce our register wiping in some places. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-20 13:15 ` Chris Snook 0 siblings, 0 replies; 333+ messages in thread From: Chris Snook @ 2007-08-20 13:15 UTC (permalink / raw) To: Linus Torvalds Cc: Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Linus Torvalds wrote: > So the only reason to add back "volatile" to the atomic_read() sequence is > not to fix bugs, but to _hide_ the bugs better. They're still there, they > are just a lot harder to trigger, and tend to be a lot subtler. What about barrier removal? With consistent semantics we could optimize a fair amount of code. Whether or not that constitutes "premature" optimization is open to debate, but there's no question we could reduce our register wiping in some places. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 13:15 ` Chris Snook (?) @ 2007-08-20 13:32 ` Herbert Xu 2007-08-20 13:38 ` Chris Snook -1 siblings, 1 reply; 333+ messages in thread From: Herbert Xu @ 2007-08-20 13:32 UTC (permalink / raw) To: Chris Snook Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, Aug 20, 2007 at 09:15:11AM -0400, Chris Snook wrote: > Linus Torvalds wrote: > >So the only reason to add back "volatile" to the atomic_read() sequence is > >not to fix bugs, but to _hide_ the bugs better. They're still there, they > >are just a lot harder to trigger, and tend to be a lot subtler. > > What about barrier removal? With consistent semantics we could optimize a > fair amount of code. Whether or not that constitutes "premature" > optimization is open to debate, but there's no question we could reduce our > register wiping in some places. If you've been reading all of Linus's emails you should be thinking about adding memory barriers, and not removing compiler barriers. He's just told you that code of the kind while (!atomic_read(cond)) ; do_something() probably needs a memory barrier (not just compiler) so that do_something() doesn't see stale cache content that occured before cond flipped. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 13:32 ` Herbert Xu @ 2007-08-20 13:38 ` Chris Snook 2007-08-20 22:07 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Chris Snook @ 2007-08-20 13:38 UTC (permalink / raw) To: Herbert Xu Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Herbert Xu wrote: > On Mon, Aug 20, 2007 at 09:15:11AM -0400, Chris Snook wrote: >> Linus Torvalds wrote: >>> So the only reason to add back "volatile" to the atomic_read() sequence is >>> not to fix bugs, but to _hide_ the bugs better. They're still there, they >>> are just a lot harder to trigger, and tend to be a lot subtler. >> What about barrier removal? With consistent semantics we could optimize a >> fair amount of code. Whether or not that constitutes "premature" >> optimization is open to debate, but there's no question we could reduce our >> register wiping in some places. > > If you've been reading all of Linus's emails you should be > thinking about adding memory barriers, and not removing > compiler barriers. > > He's just told you that code of the kind > > while (!atomic_read(cond)) > ; > > do_something() > > probably needs a memory barrier (not just compiler) so that > do_something() doesn't see stale cache content that occured > before cond flipped. Such code generally doesn't care precisely when it gets the update, just that the update is atomic, and it doesn't loop forever. Regardless, I'm convinced we just need to do it all in assembly. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 13:38 ` Chris Snook @ 2007-08-20 22:07 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-20 22:07 UTC (permalink / raw) To: Chris Snook Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, Stefan Richter, horms, Satyam Sharma, Ilpo Jarvinen, Linux Kernel Mailing List, David Miller, Paul E. McKenney, ak, Netdev, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Herbert Xu, Linus Torvalds, wensong, Nick Piggin, wjiang > Such code generally doesn't care precisely when it gets the update, > just that the update is atomic, and it doesn't loop forever. Yes, it _does_ care that it gets the update _at all_, and preferably as early as possible. > Regardless, I'm convinced we just need to do it all in assembly. So do you want "volatile asm" or "plain asm", for atomic_read()? The asm version has two ways to go about it too... Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-20 13:15 ` Chris Snook (?) (?) @ 2007-08-21 5:46 ` Linus Torvalds 2007-08-21 7:04 ` David Miller -1 siblings, 1 reply; 333+ messages in thread From: Linus Torvalds @ 2007-08-21 5:46 UTC (permalink / raw) To: Chris Snook Cc: Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, 20 Aug 2007, Chris Snook wrote: > > What about barrier removal? With consistent semantics we could optimize a > fair amount of code. Whether or not that constitutes "premature" optimization > is open to debate, but there's no question we could reduce our register wiping > in some places. Why do people think that barriers are expensive? They really aren't. Especially the regular compiler barrier is basically zero cost. Any reasonable compiler will just flush the stuff it holds in registers that isn't already automatic local variables, and for regular kernel code, that tends to basically be nothing at all. Ie a "barrier()" is likely _cheaper_ than the code generation downside from using "volatile". Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 5:46 ` Linus Torvalds @ 2007-08-21 7:04 ` David Miller 2007-08-21 13:50 ` Chris Snook 0 siblings, 1 reply; 333+ messages in thread From: David Miller @ 2007-08-21 7:04 UTC (permalink / raw) To: torvalds Cc: csnook, piggin, satyam, herbert, paulus, clameter, ilpo.jarvinen, paulmck, stefanr, linux-kernel, linux-arch, netdev, akpm, ak, heiko.carstens, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher From: Linus Torvalds <torvalds@linux-foundation.org> Date: Mon, 20 Aug 2007 22:46:47 -0700 (PDT) > Ie a "barrier()" is likely _cheaper_ than the code generation downside > from using "volatile". Assuming GCC were ever better about the code generation badness with volatile that has been discussed here, I much prefer we tell GCC "this memory piece changed" rather than "every piece of memory has changed" which is what the barrier() does. I happened to have been scanning a lot of assembler lately to track down a gcc-4.2 miscompilation on sparc64, and the barriers do hurt quite a bit in some places. Instead of keeping unrelated variables around cached in local registers, it reloads everything. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 7:04 ` David Miller @ 2007-08-21 13:50 ` Chris Snook 2007-08-21 14:59 ` Segher Boessenkool ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Chris Snook @ 2007-08-21 13:50 UTC (permalink / raw) To: David Miller Cc: torvalds, piggin, satyam, herbert, paulus, clameter, ilpo.jarvinen, paulmck, stefanr, linux-kernel, linux-arch, netdev, akpm, ak, heiko.carstens, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher David Miller wrote: > From: Linus Torvalds <torvalds@linux-foundation.org> > Date: Mon, 20 Aug 2007 22:46:47 -0700 (PDT) > >> Ie a "barrier()" is likely _cheaper_ than the code generation downside >> from using "volatile". > > Assuming GCC were ever better about the code generation badness > with volatile that has been discussed here, I much prefer > we tell GCC "this memory piece changed" rather than "every > piece of memory has changed" which is what the barrier() does. > > I happened to have been scanning a lot of assembler lately to > track down a gcc-4.2 miscompilation on sparc64, and the barriers > do hurt quite a bit in some places. Instead of keeping unrelated > variables around cached in local registers, it reloads everything. Moore's law is definitely working against us here. Register counts, pipeline depths, core counts, and clock multipliers are all increasing in the long run. At some point in the future, barrier() will be universally regarded as a hammer too big for most purposes. Whether or not removing it now constitutes premature optimization is arguable, but I think we should allow such optimization to happen (or not happen) in architecture-dependent code, and provide a consistent API that doesn't require the use of such things in arch-independent code where it might turn into a totally superfluous performance killer depending on what hardware it gets compiled for. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 13:50 ` Chris Snook @ 2007-08-21 14:59 ` Segher Boessenkool 2007-08-21 16:31 ` Satyam Sharma 2007-08-21 16:43 ` Linus Torvalds 2 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-21 14:59 UTC (permalink / raw) To: Chris Snook Cc: paulmck, heiko.carstens, ilpo.jarvinen, horms, linux-kernel, David Miller, rpjday, netdev, ak, piggin, akpm, torvalds, cfriesen, jesper.juhl, linux-arch, paulus, herbert, satyam, clameter, stefanr, schwidefsky, zlynx, wensong, wjiang > At some point in the future, barrier() will be universally regarded as > a hammer too big for most purposes. Whether or not removing it now You can't just remove it, it is needed in some places; you want to replace it in most places with a more fine-grained "compiler barrier", I presume? > constitutes premature optimization is arguable, but I think we should > allow such optimization to happen (or not happen) in > architecture-dependent code, and provide a consistent API that doesn't > require the use of such things in arch-independent code where it might > turn into a totally superfluous performance killer depending on what > hardware it gets compiled for. Explicit barrier()s won't be too hard to replace -- but what to do about the implicit barrier()s in rmb() etc. etc. -- *those* will be hard to get rid of, if only because it is hard enough to teach driver authors about how to use those primitives *already*. It is far from clear what a good interface like that would look like, anyway. Probably we should first start experimenting with a forget()-style micro-barrier (but please, find a better name), and see if a nice usage pattern shows up that can be turned into an API. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 13:50 ` Chris Snook 2007-08-21 14:59 ` Segher Boessenkool @ 2007-08-21 16:31 ` Satyam Sharma 2007-08-21 16:43 ` Linus Torvalds 2 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-21 16:31 UTC (permalink / raw) To: Chris Snook Cc: David Miller, Linus Torvalds, piggin, herbert, paulus, clameter, ilpo.jarvinen, paulmck, stefanr, Linux Kernel Mailing List, linux-arch, netdev, Andrew Morton, ak, heiko.carstens, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Tue, 21 Aug 2007, Chris Snook wrote: > David Miller wrote: > > From: Linus Torvalds <torvalds@linux-foundation.org> > > Date: Mon, 20 Aug 2007 22:46:47 -0700 (PDT) > > > > > Ie a "barrier()" is likely _cheaper_ than the code generation downside > > > from using "volatile". > > > > Assuming GCC were ever better about the code generation badness > > with volatile that has been discussed here, I much prefer > > we tell GCC "this memory piece changed" rather than "every > > piece of memory has changed" which is what the barrier() does. > > > > I happened to have been scanning a lot of assembler lately to > > track down a gcc-4.2 miscompilation on sparc64, and the barriers > > do hurt quite a bit in some places. Instead of keeping unrelated > > variables around cached in local registers, it reloads everything. > > Moore's law is definitely working against us here. Register counts, pipeline > depths, core counts, and clock multipliers are all increasing in the long run. > At some point in the future, barrier() will be universally regarded as a > hammer too big for most purposes. I do agree, and the important point to note is that the benefits of a /lighter/ compiler barrier, such as what David referred to above, _can_ be had without having to do anything with the "volatile" keyword at all. And such a primitive has already been mentioned/proposed on this thread. But this is all tangential to the core question at hand -- whether to have implicit (compiler, possibly "light-weight" of the kind referred above) barrier semantics in atomic ops that do not have them, or not. I was lately looking in the kernel for _actual_ code that uses atomic_t and benefits from the lack of any implicit barrier, with the compiler being free to cache the atomic_t in a register. Now that often does _not_ happen, because all other ops (implemented in asm with LOCK prefix on x86) _must_ therefore constrain the atomic_t to memory anyway. So typically all atomic ops code sequences end up operating on memory. Then I did locate sched.c:select_nohz_load_balancer() -- it repeatedly references the same atomic_t object, and the code that I saw generated (with CC_OPTIMIZE_FOR_SIZE=y) did cache it in a register for a sequence of instructions. It uses atomic_cmpxchg, thereby not requiring explicit memory barriers anywhere in the code, and is an example of an atomic_t user that is safe, and yet benefits from its memory loads/stores being elided/coalesced by the compiler. # at this point, %%eax holds num_online_cpus() and # %%ebx holds cpus_weight(nohz.cpu_mask) # the variable "cpu" is in %esi 0xc1018e1d: cmp %eax,%ebx # if No.A. 0xc1018e1f: mov 0xc134d900,%eax # first atomic_read() 0xc1018e24: jne 0xc1018e36 0xc1018e26: cmp %esi,%eax # if No.B. 0xc1018e28: jne 0xc1018e80 # returns with 0 0xc1018e2a: movl $0xffffffff,0xc134d900 # atomic_set(-1), and ... 0xc1018e34: jmp 0xc1018e80 # ... returns with 0 0xc1018e36: cmp $0xffffffff,%eax # if No.C. (NOTE!) 0xc1018e39: jne 0xc1018e46 0xc1018e3b: lock cmpxchg %esi,0xc134d900 # atomic_cmpxchg() 0xc1018e43: inc %eax 0xc1018e44: jmp 0xc1018e48 0xc1018e46: cmp %esi,%eax # if No.D. (NOTE!) 0xc1018e48: jne 0xc1018e80 # if !=, default return 0 (if No.E.) 0xc1018e4a: jmp 0xc1018e84 # otherwise (==) returns with 1 The above is: if (cpus_weight(nohz.cpu_mask) == num_online_cpus()) { /* if No.A. */ if (atomic_read(&nohz.load_balancer) == cpu) /* if No.B. */ atomic_set(&nohz.load_balancer, -1); /* XXX */ return 0; } if (atomic_read(&nohz.load_balancer) == -1) { /* if No.C. */ /* make me the ilb owner */ if (atomic_cmpxchg(&nohz.load_balancer, -1, cpu) == -1) /* if No.E. */ return 1; } else if (atomic_read(&nohz.load_balancer) == cpu) /* if No.D. */ return 1; ... ... return 0; /* default return from function */ As you can see, the atomic_read()'s of "if"s Nos. B, C, and D, were _all_ coalesced into a single memory reference "mov 0xc134d900,%eax" at the top of the function, and then "if"s Nos. C and D simply used the value from %%eax itself. But that's perfectly safe, such is the logic of this function. It uses cmpxchg _whenever_ updating the value in the memory atomic_t and then returns appropriately. The _only_ point that a casual reader may find racy is that marked /* XXX */ above -- atomic_read() followed by atomic_set() with no barrier in between. But even that is ok, because if one thread ever finds that condition to succeed, it is 100% guaranteed no other thread on any other CPU will find _any_ condition to be true, thereby avoiding any race in the modification of that value. BTW it does sound reasonable that a lot of atomic_t users that want a compiler barrier probably also want a memory barrier. Do we make _that_ implicit too? Quite clearly, making _either_ one of those implicit in atomic_{read,set} (in any form of implementation -- a forget() macro based, *(volatile int *)& based, or inline asm based) would end up harming code such as that cited above. Lastly, the most obvious reason that should be considered against implicit barriers in atomic ops is that it isn't "required" -- atomicity does not imply any barrier after all, and making such a distinction would actually be a healthy separation that helps people think more clearly when writing lockless code. [ But the "authors' expectations" / heisenbugs argument also holds some water ... for that, we can have a _variant_ in the API for atomic ops that has implicit compiler/memory barriers, to make it easier on those who want that behaviour. But let us not penalize code that knows what it is doing by changing the default to that, please. ] Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-21 13:50 ` Chris Snook 2007-08-21 14:59 ` Segher Boessenkool 2007-08-21 16:31 ` Satyam Sharma @ 2007-08-21 16:43 ` Linus Torvalds 2 siblings, 0 replies; 333+ messages in thread From: Linus Torvalds @ 2007-08-21 16:43 UTC (permalink / raw) To: Chris Snook Cc: David Miller, piggin, satyam, herbert, paulus, clameter, ilpo.jarvinen, paulmck, stefanr, linux-kernel, linux-arch, netdev, akpm, ak, heiko.carstens, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Tue, 21 Aug 2007, Chris Snook wrote: > > Moore's law is definitely working against us here. Register counts, pipeline > depths, core counts, and clock multipliers are all increasing in the long run. > At some point in the future, barrier() will be universally regarded as a > hammer too big for most purposes. Note that "barrier()" is purely a compiler barrier. It has zero impact on the CPU pipeline itself, and also has zero impact on anything that gcc knows isn't visible in memory (ie local variables that don't have their address taken), so barrier() really is pretty cheap. Now, it's possible that gcc messes up in some circumstances, and that the memory clobber will cause gcc to also do things like flush local registers unnecessarily to their stack slots, but quite frankly, if that happens, it's a gcc problem, and I also have to say that I've not seen that myself. So in a very real sense, "barrier()" will just make sure that there is a stronger sequence point for the compiler where things are stable. In most cases it has absolutely zero performance impact - apart from the -intended- impact of making sure that the compiler doesn't re-order or cache stuff around it. And sure, we could make it more finegrained, and also introduce a per-variable barrier, but the fact is, people _already_ have problems with thinking about these kinds of things, and adding new abstraction issues with subtle semantics is the last thing we want. So I really think you'd want to show a real example of real code that actually gets noticeably slower or bigger. In removing "volatile", we have shown that. It may not have made a big difference on powerpc, but it makes a real difference on x86 - and more importantly, it removes something that people clearly don't know how it works, and incorrectly expect to just fix bugs. [ There are *other* barriers - the ones that actually add memory barriers to the CPU - that really can be quite expensive. The good news is that the expense is going down rather than up: both Intel and AMD are not only removing the need for some of them (ie "smp_rmb()" will become a compiler-only barrier), but we're _also_ seeing the whole "pipeline flush" approach go away, and be replaced by the CPU itself actually being better - so even the actual CPU pipeline barriers are getting cheaper, not more expensive. ] For example, did anybody even _test_ how expensive "barrier()" is? Just as a lark, I did #undef barrier #define barrier() do { } while (0) in kernel/sched.c (which only has three of them in it, but hey, that's more than most files), and there were _zero_ code generation downsides. One instruction was moved (and a few line numbers changed), so it wasn't like the assembly language was identical, but the point is, barrier() simply doesn't have the same kinds of downsides that "volatile" has. (That may not be true on other architectures or in other source files, of course. This *does* depend on code generation details. But anybody who thinks that "barrier()" is fundamentally expensive is simply incorrect. It is *fundamnetally* a no-op). Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 16:48 ` Linus Torvalds 2007-08-17 18:50 ` Chris Friesen 2007-08-20 13:15 ` Chris Snook @ 2007-09-09 18:02 ` Denys Vlasenko 2007-09-09 18:18 ` Arjan van de Ven 2 siblings, 1 reply; 333+ messages in thread From: Denys Vlasenko @ 2007-09-09 18:02 UTC (permalink / raw) To: Linus Torvalds Cc: Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Friday 17 August 2007 17:48, Linus Torvalds wrote: > > On Fri, 17 Aug 2007, Nick Piggin wrote: > > > > That's not obviously just taste to me. Not when the primitive has many > > (perhaps, the majority) of uses that do not require said barriers. And > > this is not solely about the code generation (which, as Paul says, is > > relatively minor even on x86). I prefer people to think explicitly > > about barriers in their lockless code. > > Indeed. > > I think the important issues are: > > - "volatile" itself is simply a badly/weakly defined issue. The semantics > of it as far as the compiler is concerned are really not very good, and > in practice tends to boil down to "I will generate so bad code that > nobody can accuse me of optimizing anything away". > > - "volatile" - regardless of how well or badly defined it is - is purely > a compiler thing. It has absolutely no meaning for the CPU itself, so > it at no point implies any CPU barriers. As a result, even if the > compiler generates crap code and doesn't re-order anything, there's > nothing that says what the CPU will do. > > - in other words, the *only* possible meaning for "volatile" is a purely > single-CPU meaning. And if you only have a single CPU involved in the > process, the "volatile" is by definition pointless (because even > without a volatile, the compiler is required to make the C code appear > consistent as far as a single CPU is concerned). > > So, let's take the example *buggy* code where we use "volatile" to wait > for other CPU's: > > atomic_set(&var, 0); > while (!atomic_read(&var)) > /* nothing */; > > > which generates an endless loop if we don't have atomic_read() imply > volatile. > > The point here is that it's buggy whether the volatile is there or not! > Exactly because the user expects multi-processing behaviour, but > "volatile" doesn't actually give any real guarantees about it. Another CPU > may have done: > > external_ptr = kmalloc(..); > /* Setup is now complete, inform the waiter */ > atomic_inc(&var); > > but the fact is, since the other CPU isn't serialized in any way, the > "while-loop" (even in the presense of "volatile") doesn't actually work > right! Whatever the "atomic_read()" was waiting for may not have > completed, because we have no barriers! Why is all this fixation on "volatile"? I don't think people want "volatile" keyword per se, they want atomic_read(&x) to _always_ compile into an memory-accessing instruction, not register access. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-09 18:02 ` Denys Vlasenko @ 2007-09-09 18:18 ` Arjan van de Ven 2007-09-10 10:56 ` Denys Vlasenko 0 siblings, 1 reply; 333+ messages in thread From: Arjan van de Ven @ 2007-09-09 18:18 UTC (permalink / raw) To: Denys Vlasenko Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Sun, 9 Sep 2007 19:02:54 +0100 Denys Vlasenko <vda.linux@googlemail.com> wrote: > Why is all this fixation on "volatile"? I don't think > people want "volatile" keyword per se, they want atomic_read(&x) to > _always_ compile into an memory-accessing instruction, not register > access. and ... why is that? is there any valid, non-buggy code sequence that makes that a reasonable requirement? ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-09 18:18 ` Arjan van de Ven @ 2007-09-10 10:56 ` Denys Vlasenko 2007-09-10 11:15 ` Herbert Xu ` (2 more replies) 0 siblings, 3 replies; 333+ messages in thread From: Denys Vlasenko @ 2007-09-10 10:56 UTC (permalink / raw) To: Arjan van de Ven Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Sunday 09 September 2007 19:18, Arjan van de Ven wrote: > On Sun, 9 Sep 2007 19:02:54 +0100 > Denys Vlasenko <vda.linux@googlemail.com> wrote: > > > Why is all this fixation on "volatile"? I don't think > > people want "volatile" keyword per se, they want atomic_read(&x) to > > _always_ compile into an memory-accessing instruction, not register > > access. > > and ... why is that? > is there any valid, non-buggy code sequence that makes that a > reasonable requirement? Well, if you insist on having it again: Waiting for atomic value to be zero: while (atomic_read(&x)) continue; gcc may happily convert it into: reg = atomic_read(&x); while (reg) continue; Expecting every driver writer to remember that atomic_read is not in fact a "read from memory" is naive. That won't happen. Face it, majority of driver authors are a bit less talented than Ingo Molnar or Arjan van de Ven ;) The name of the macro is saying that it's a read. We are confusing users here. It's doubly confusing that cpy_relax(), which says _nothing_ about barriers in its name, is actually a barrier you need to insert here. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 10:56 ` Denys Vlasenko @ 2007-09-10 11:15 ` Herbert Xu 2007-09-10 12:22 ` Kyle Moffett 2007-09-10 14:51 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Arjan van de Ven 2 siblings, 0 replies; 333+ messages in thread From: Herbert Xu @ 2007-09-10 11:15 UTC (permalink / raw) To: Denys Vlasenko Cc: Arjan van de Ven, Linus Torvalds, Nick Piggin, Satyam Sharma, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, Sep 10, 2007 at 11:56:29AM +0100, Denys Vlasenko wrote: > > Expecting every driver writer to remember that atomic_read is not in fact > a "read from memory" is naive. That won't happen. Face it, majority of > driver authors are a bit less talented than Ingo Molnar or Arjan van de Ven ;) > The name of the macro is saying that it's a read. > We are confusing users here. For driver authors who're too busy to learn the intricacies of atomic operations, we have the plain old spin lock which then lets you use normal data structures such as u32 safely. Cheers, -- Visit Openswan at http://www.openswan.org/ Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 10:56 ` Denys Vlasenko @ 2007-09-10 12:22 ` Kyle Moffett 2007-09-10 12:22 ` Kyle Moffett 2007-09-10 14:51 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Arjan van de Ven 2 siblings, 0 replies; 333+ messages in thread From: Kyle Moffett @ 2007-09-10 12:22 UTC (permalink / raw) To: Denys Vlasenko Cc: Arjan van de Ven, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Sep 10, 2007, at 06:56:29, Denys Vlasenko wrote: > On Sunday 09 September 2007 19:18, Arjan van de Ven wrote: >> On Sun, 9 Sep 2007 19:02:54 +0100 >> Denys Vlasenko <vda.linux@googlemail.com> wrote: >> >>> Why is all this fixation on "volatile"? I don't think people want >>> "volatile" keyword per se, they want atomic_read(&x) to _always_ >>> compile into an memory-accessing instruction, not register access. >> >> and ... why is that? is there any valid, non-buggy code sequence >> that makes that a reasonable requirement? > > Well, if you insist on having it again: > > Waiting for atomic value to be zero: > > while (atomic_read(&x)) > continue; > > gcc may happily convert it into: > > reg = atomic_read(&x); > while (reg) > continue; Bzzt. Even if you fixed gcc to actually convert it to a busy loop on a memory variable, you STILL HAVE A BUG as it may *NOT* be gcc that does the conversion, it may be that the CPU does the caching of the memory value. GCC has no mechanism to do cache-flushes or memory- barriers except through our custom inline assembly. Also, you probably want a cpu_relax() in there somewhere to avoid overheating the CPU. Thirdly, on a large system it may take some arbitrarily large amount of time for cache-propagation to update the value of the variable in your local CPU cache. Finally, if atomics are based on based on spinlock+interrupt-disable then you will sit in a tight busy- loop of spin_lock_irqsave()->spin_unlock_irqrestore(). Depending on your system's internal model this may practically lock up your core because the spin_lock() will take the cacheline for exclusive access and doing that in a loop can prevent any other CPU from doing any operation on it! Since your IRQs are disabled you even have a very small window that an IRQ will come along and free it up long enough for the update to take place. The earlier code segment of: > while(atomic_read(&x) > 0) > atomic_dec(&x); is *completely* buggy because you could very easily have 4 CPUs doing this on an atomic variable with a value of 1 and end up with it at negative 3 by the time you are done. Moreover all the alternatives are also buggy, with the sole exception of this rather obvious- seeming one: > atomic_set(&x, 0); You simply CANNOT use an atomic_t as your sole synchronizing primitive, it doesn't work! You virtually ALWAYS want to use an atomic_t in the following types of situations: (A) As an object refcount. The value is never read except as part of an atomic_dec_return(). Why aren't you using "struct kref"? (B) As an atomic value counter (number of processes, for example). Just "reading" the value is racy anyways, if you want to enforce a limit or something then use atomic_inc_return(), check the result, and use atomic_dec() if it's too big. If you just want to return the statistics then you are going to be instantaneous-point-in-time anyways. (C) As an optimization value (statistics-like, but exact accuracy isn't important). Atomics are NOT A REPLACEMENT for the proper kernel subsystem, like completions, mutexes, semaphores, spinlocks, krefs, etc. It's not useful for synchronization, only for keeping track of simple integer RMW values. Note that atomic_read() and atomic_set() aren't very useful RMW primitives (read-nomodify-nowrite and read-set-zero- write). Code which assumes anything else is probably buggy in other ways too. So while I see no real reason for the "volatile" on the atomics, I also see no real reason why it's terribly harmful. Regardless of the "volatile" on the operation the CPU is perfectly happy to cache it anyways so it doesn't buy you any actual "always-access-memory" guarantees. If you are just interested in it as an optimization you could probably just read the properly-aligned integer counter directly, an atomic read on most CPUs. If you really need it to hit main memory *every* *single* *time* (Why? Are you using it instead of the proper kernel subsystem?) then you probably need a custom inline assembly helper anyways. Cheers, Kyle Moffett ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-09-10 12:22 ` Kyle Moffett 0 siblings, 0 replies; 333+ messages in thread From: Kyle Moffett @ 2007-09-10 12:22 UTC (permalink / raw) To: Denys Vlasenko Cc: Arjan van de Ven, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Sep 10, 2007, at 06:56:29, Denys Vlasenko wrote: > On Sunday 09 September 2007 19:18, Arjan van de Ven wrote: >> On Sun, 9 Sep 2007 19:02:54 +0100 >> Denys Vlasenko <vda.linux@googlemail.com> wrote: >> >>> Why is all this fixation on "volatile"? I don't think people want >>> "volatile" keyword per se, they want atomic_read(&x) to _always_ >>> compile into an memory-accessing instruction, not register access. >> >> and ... why is that? is there any valid, non-buggy code sequence >> that makes that a reasonable requirement? > > Well, if you insist on having it again: > > Waiting for atomic value to be zero: > > while (atomic_read(&x)) > continue; > > gcc may happily convert it into: > > reg = atomic_read(&x); > while (reg) > continue; Bzzt. Even if you fixed gcc to actually convert it to a busy loop on a memory variable, you STILL HAVE A BUG as it may *NOT* be gcc that does the conversion, it may be that the CPU does the caching of the memory value. GCC has no mechanism to do cache-flushes or memory- barriers except through our custom inline assembly. Also, you probably want a cpu_relax() in there somewhere to avoid overheating the CPU. Thirdly, on a large system it may take some arbitrarily large amount of time for cache-propagation to update the value of the variable in your local CPU cache. Finally, if atomics are based on based on spinlock+interrupt-disable then you will sit in a tight busy- loop of spin_lock_irqsave()->spin_unlock_irqrestore(). Depending on your system's internal model this may practically lock up your core because the spin_lock() will take the cacheline for exclusive access and doing that in a loop can prevent any other CPU from doing any operation on it! Since your IRQs are disabled you even have a very small window that an IRQ will come along and free it up long enough for the update to take place. The earlier code segment of: > while(atomic_read(&x) > 0) > atomic_dec(&x); is *completely* buggy because you could very easily have 4 CPUs doing this on an atomic variable with a value of 1 and end up with it at negative 3 by the time you are done. Moreover all the alternatives are also buggy, with the sole exception of this rather obvious- seeming one: > atomic_set(&x, 0); You simply CANNOT use an atomic_t as your sole synchronizing primitive, it doesn't work! You virtually ALWAYS want to use an atomic_t in the following types of situations: (A) As an object refcount. The value is never read except as part of an atomic_dec_return(). Why aren't you using "struct kref"? (B) As an atomic value counter (number of processes, for example). Just "reading" the value is racy anyways, if you want to enforce a limit or something then use atomic_inc_return(), check the result, and use atomic_dec() if it's too big. If you just want to return the statistics then you are going to be instantaneous-point-in-time anyways. (C) As an optimization value (statistics-like, but exact accuracy isn't important). Atomics are NOT A REPLACEMENT for the proper kernel subsystem, like completions, mutexes, semaphores, spinlocks, krefs, etc. It's not useful for synchronization, only for keeping track of simple integer RMW values. Note that atomic_read() and atomic_set() aren't very useful RMW primitives (read-nomodify-nowrite and read-set-zero- write). Code which assumes anything else is probably buggy in other ways too. So while I see no real reason for the "volatile" on the atomics, I also see no real reason why it's terribly harmful. Regardless of the "volatile" on the operation the CPU is perfectly happy to cache it anyways so it doesn't buy you any actual "always-access-memory" guarantees. If you are just interested in it as an optimization you could probably just read the properly-aligned integer counter directly, an atomic read on most CPUs. If you really need it to hit main memory *every* *single* *time* (Why? Are you using it instead of the proper kernel subsystem?) then you probably need a custom inline assembly helper anyways. Cheers, Kyle Moffett ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 12:22 ` Kyle Moffett @ 2007-09-10 13:38 ` Denys Vlasenko -1 siblings, 0 replies; 333+ messages in thread From: Denys Vlasenko @ 2007-09-10 13:38 UTC (permalink / raw) To: Kyle Moffett Cc: Arjan van de Ven, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Monday 10 September 2007 13:22, Kyle Moffett wrote: > On Sep 10, 2007, at 06:56:29, Denys Vlasenko wrote: > > On Sunday 09 September 2007 19:18, Arjan van de Ven wrote: > >> On Sun, 9 Sep 2007 19:02:54 +0100 > >> Denys Vlasenko <vda.linux@googlemail.com> wrote: > >> > >>> Why is all this fixation on "volatile"? I don't think people want > >>> "volatile" keyword per se, they want atomic_read(&x) to _always_ > >>> compile into an memory-accessing instruction, not register access. > >> > >> and ... why is that? is there any valid, non-buggy code sequence > >> that makes that a reasonable requirement? > > > > Well, if you insist on having it again: > > > > Waiting for atomic value to be zero: > > > > while (atomic_read(&x)) > > continue; > > > > gcc may happily convert it into: > > > > reg = atomic_read(&x); > > while (reg) > > continue; > > Bzzt. Even if you fixed gcc to actually convert it to a busy loop on > a memory variable, you STILL HAVE A BUG as it may *NOT* be gcc that > does the conversion, it may be that the CPU does the caching of the > memory value. GCC has no mechanism to do cache-flushes or memory- > barriers except through our custom inline assembly. CPU can cache the value all right, but it cannot use that cached value *forever*, it has to react to invalidate cycles on the shared bus and re-fetch new data. IOW: atomic_read(&x) which compiles down to memory accessor will work properly. > the CPU. Thirdly, on a large system it may take some arbitrarily > large amount of time for cache-propagation to update the value of the > variable in your local CPU cache. Yes, but "arbitrarily large amount of time" is actually measured in nanoseconds here. Let's say 1000ns max for hundreds of CPUs? > Also, you > probably want a cpu_relax() in there somewhere to avoid overheating > the CPU. Yes, but 1. CPU shouldn't overheat (in a sense that it gets damaged), it will only use more power than needed. 2. cpu_relax() just throttles down my CPU, so it's performance optimization only. Wait, it isn't, it's a barrier too. Wow, "cpu_relax" is a barrier? How am I supposed to know that without reading lkml flamewars and/or header files? Let's try reading headers. asm-x86_64/processor.h: #define cpu_relax() rep_nop() So, is it a barrier? No clue yet. /* REP NOP (PAUSE) is a good thing to insert into busy-wait loops. */ static inline void rep_nop(void) { __asm__ __volatile__("rep;nop": : :"memory"); } Comment explicitly says that it is "a good thing" (doesn't say that it is mandatory) and says NOTHING about barriers! Barrier-ness is not mentioned and is hidden in "memory" clobber. Do you think it's obvious enough for average driver writer? I think not, especially that it's unlikely for him to even start suspecting that it is a memory barrier based on the "cpu_relax" name. > You simply CANNOT use an atomic_t as your sole synchronizing > primitive, it doesn't work! You virtually ALWAYS want to use an > atomic_t in the following types of situations: > > (A) As an object refcount. The value is never read except as part of > an atomic_dec_return(). Why aren't you using "struct kref"? > > (B) As an atomic value counter (number of processes, for example). > Just "reading" the value is racy anyways, if you want to enforce a > limit or something then use atomic_inc_return(), check the result, > and use atomic_dec() if it's too big. If you just want to return the > statistics then you are going to be instantaneous-point-in-time anyways. > > (C) As an optimization value (statistics-like, but exact accuracy > isn't important). > > Atomics are NOT A REPLACEMENT for the proper kernel subsystem, like > completions, mutexes, semaphores, spinlocks, krefs, etc. It's not > useful for synchronization, only for keeping track of simple integer > RMW values. Note that atomic_read() and atomic_set() aren't very > useful RMW primitives (read-nomodify-nowrite and read-set-zero- > write). Code which assumes anything else is probably buggy in other > ways too. You are basically trying to educate me how to use atomic properly. You don't need to do it, as I am (currently) not a driver author. I am saying that people who are already using atomic_read() (and who unfortunately did not read your explanation above) will still sometimes use atomic_read() as a way to read atomic value *from memory*, and will create nasty heisenbugs for you to debug. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-09-10 13:38 ` Denys Vlasenko 0 siblings, 0 replies; 333+ messages in thread From: Denys Vlasenko @ 2007-09-10 13:38 UTC (permalink / raw) To: Kyle Moffett Cc: Arjan van de Ven, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Monday 10 September 2007 13:22, Kyle Moffett wrote: > On Sep 10, 2007, at 06:56:29, Denys Vlasenko wrote: > > On Sunday 09 September 2007 19:18, Arjan van de Ven wrote: > >> On Sun, 9 Sep 2007 19:02:54 +0100 > >> Denys Vlasenko <vda.linux@googlemail.com> wrote: > >> > >>> Why is all this fixation on "volatile"? I don't think people want > >>> "volatile" keyword per se, they want atomic_read(&x) to _always_ > >>> compile into an memory-accessing instruction, not register access. > >> > >> and ... why is that? is there any valid, non-buggy code sequence > >> that makes that a reasonable requirement? > > > > Well, if you insist on having it again: > > > > Waiting for atomic value to be zero: > > > > while (atomic_read(&x)) > > continue; > > > > gcc may happily convert it into: > > > > reg = atomic_read(&x); > > while (reg) > > continue; > > Bzzt. Even if you fixed gcc to actually convert it to a busy loop on > a memory variable, you STILL HAVE A BUG as it may *NOT* be gcc that > does the conversion, it may be that the CPU does the caching of the > memory value. GCC has no mechanism to do cache-flushes or memory- > barriers except through our custom inline assembly. CPU can cache the value all right, but it cannot use that cached value *forever*, it has to react to invalidate cycles on the shared bus and re-fetch new data. IOW: atomic_read(&x) which compiles down to memory accessor will work properly. > the CPU. Thirdly, on a large system it may take some arbitrarily > large amount of time for cache-propagation to update the value of the > variable in your local CPU cache. Yes, but "arbitrarily large amount of time" is actually measured in nanoseconds here. Let's say 1000ns max for hundreds of CPUs? > Also, you > probably want a cpu_relax() in there somewhere to avoid overheating > the CPU. Yes, but 1. CPU shouldn't overheat (in a sense that it gets damaged), it will only use more power than needed. 2. cpu_relax() just throttles down my CPU, so it's performance optimization only. Wait, it isn't, it's a barrier too. Wow, "cpu_relax" is a barrier? How am I supposed to know that without reading lkml flamewars and/or header files? Let's try reading headers. asm-x86_64/processor.h: #define cpu_relax() rep_nop() So, is it a barrier? No clue yet. /* REP NOP (PAUSE) is a good thing to insert into busy-wait loops. */ static inline void rep_nop(void) { __asm__ __volatile__("rep;nop": : :"memory"); } Comment explicitly says that it is "a good thing" (doesn't say that it is mandatory) and says NOTHING about barriers! Barrier-ness is not mentioned and is hidden in "memory" clobber. Do you think it's obvious enough for average driver writer? I think not, especially that it's unlikely for him to even start suspecting that it is a memory barrier based on the "cpu_relax" name. > You simply CANNOT use an atomic_t as your sole synchronizing > primitive, it doesn't work! You virtually ALWAYS want to use an > atomic_t in the following types of situations: > > (A) As an object refcount. The value is never read except as part of > an atomic_dec_return(). Why aren't you using "struct kref"? > > (B) As an atomic value counter (number of processes, for example). > Just "reading" the value is racy anyways, if you want to enforce a > limit or something then use atomic_inc_return(), check the result, > and use atomic_dec() if it's too big. If you just want to return the > statistics then you are going to be instantaneous-point-in-time anyways. > > (C) As an optimization value (statistics-like, but exact accuracy > isn't important). > > Atomics are NOT A REPLACEMENT for the proper kernel subsystem, like > completions, mutexes, semaphores, spinlocks, krefs, etc. It's not > useful for synchronization, only for keeping track of simple integer > RMW values. Note that atomic_read() and atomic_set() aren't very > useful RMW primitives (read-nomodify-nowrite and read-set-zero- > write). Code which assumes anything else is probably buggy in other > ways too. You are basically trying to educate me how to use atomic properly. You don't need to do it, as I am (currently) not a driver author. I am saying that people who are already using atomic_read() (and who unfortunately did not read your explanation above) will still sometimes use atomic_read() as a way to read atomic value *from memory*, and will create nasty heisenbugs for you to debug. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 13:38 ` Denys Vlasenko @ 2007-09-10 14:16 ` Denys Vlasenko -1 siblings, 0 replies; 333+ messages in thread From: Denys Vlasenko @ 2007-09-10 14:16 UTC (permalink / raw) To: Kyle Moffett Cc: Arjan van de Ven, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Monday 10 September 2007 14:38, Denys Vlasenko wrote: > You are basically trying to educate me how to use atomic properly. > You don't need to do it, as I am (currently) not a driver author. > > I am saying that people who are already using atomic_read() > (and who unfortunately did not read your explanation above) > will still sometimes use atomic_read() as a way to read atomic value > *from memory*, and will create nasty heisenbugs for you to debug. static inline int qla2x00_wait_for_loop_ready(scsi_qla_host_t *ha) { int return_status = QLA_SUCCESS; unsigned long loop_timeout ; scsi_qla_host_t *pha = to_qla_parent(ha); /* wait for 5 min at the max for loop to be ready */ loop_timeout = jiffies + (MAX_LOOP_TIMEOUT * HZ); while ((!atomic_read(&pha->loop_down_timer) && atomic_read(&pha->loop_state) == LOOP_DOWN) || atomic_read(&pha->loop_state) != LOOP_READY) { if (atomic_read(&pha->loop_state) == LOOP_DEAD) { return_status = QLA_FUNCTION_FAILED; break; } msleep(1000); if (time_after_eq(jiffies, loop_timeout)) { return_status = QLA_FUNCTION_FAILED; break; } } return (return_status); } Is above correct or buggy? Correct, because msleep is a barrier. Is it obvious? No. static void qla2x00_rst_aen(scsi_qla_host_t *ha) { if (ha->flags.online && !ha->flags.reset_active && !atomic_read(&ha->loop_down_timer) && !(test_bit(ABORT_ISP_ACTIVE, &ha->dpc_flags))) { do { clear_bit(RESET_MARKER_NEEDED, &ha->dpc_flags); /* * Issue marker command only when we are going to start * the I/O. */ ha->marker_needed = 1; } while (!atomic_read(&ha->loop_down_timer) && (test_bit(RESET_MARKER_NEEDED, &ha->dpc_flags))); } } Is above correct? I honestly don't know. Correct, because set_bit is a barrier on _all _memory_? Will it break if set_bit will be changed to be a barrier only on its operand? Probably yes. drivers/kvm/kvm_main.c while (atomic_read(&completed) != needed) { cpu_relax(); barrier(); } Obviously author did not know that cpu_relax is already a barrier. See why I think driver authors will be confused? arch/x86_64/kernel/crash.c static void nmi_shootdown_cpus(void) { ... msecs = 1000; /* Wait at most a second for the other cpus to stop */ while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { mdelay(1); msecs--; } ... } Is mdelay(1) a barrier? Yes, because it is a function on x86_64. Absolutely the same code will be buggy on an arch where mdelay(1) == udelay(1000), and udelay is implemented as inline busy-wait. arch/sparc64/kernel/smp.c /* Wait for response */ while (atomic_read(&data.finished) != cpus) cpu_relax(); ...later in the same file... while (atomic_read(&smp_capture_registry) != ncpus) rmb(); I'm confused. Do we need cpu_relax() or rmb()? Does cpu_relax() imply rmb()? (No it doesn't). Which of those two while loops needs correcting? -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-09-10 14:16 ` Denys Vlasenko 0 siblings, 0 replies; 333+ messages in thread From: Denys Vlasenko @ 2007-09-10 14:16 UTC (permalink / raw) To: Kyle Moffett Cc: Arjan van de Ven, Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Monday 10 September 2007 14:38, Denys Vlasenko wrote: > You are basically trying to educate me how to use atomic properly. > You don't need to do it, as I am (currently) not a driver author. > > I am saying that people who are already using atomic_read() > (and who unfortunately did not read your explanation above) > will still sometimes use atomic_read() as a way to read atomic value > *from memory*, and will create nasty heisenbugs for you to debug. static inline int qla2x00_wait_for_loop_ready(scsi_qla_host_t *ha) { int return_status = QLA_SUCCESS; unsigned long loop_timeout ; scsi_qla_host_t *pha = to_qla_parent(ha); /* wait for 5 min at the max for loop to be ready */ loop_timeout = jiffies + (MAX_LOOP_TIMEOUT * HZ); while ((!atomic_read(&pha->loop_down_timer) && atomic_read(&pha->loop_state) == LOOP_DOWN) || atomic_read(&pha->loop_state) != LOOP_READY) { if (atomic_read(&pha->loop_state) == LOOP_DEAD) { return_status = QLA_FUNCTION_FAILED; break; } msleep(1000); if (time_after_eq(jiffies, loop_timeout)) { return_status = QLA_FUNCTION_FAILED; break; } } return (return_status); } Is above correct or buggy? Correct, because msleep is a barrier. Is it obvious? No. static void qla2x00_rst_aen(scsi_qla_host_t *ha) { if (ha->flags.online && !ha->flags.reset_active && !atomic_read(&ha->loop_down_timer) && !(test_bit(ABORT_ISP_ACTIVE, &ha->dpc_flags))) { do { clear_bit(RESET_MARKER_NEEDED, &ha->dpc_flags); /* * Issue marker command only when we are going to start * the I/O. */ ha->marker_needed = 1; } while (!atomic_read(&ha->loop_down_timer) && (test_bit(RESET_MARKER_NEEDED, &ha->dpc_flags))); } } Is above correct? I honestly don't know. Correct, because set_bit is a barrier on _all _memory_? Will it break if set_bit will be changed to be a barrier only on its operand? Probably yes. drivers/kvm/kvm_main.c while (atomic_read(&completed) != needed) { cpu_relax(); barrier(); } Obviously author did not know that cpu_relax is already a barrier. See why I think driver authors will be confused? arch/x86_64/kernel/crash.c static void nmi_shootdown_cpus(void) { ... msecs = 1000; /* Wait at most a second for the other cpus to stop */ while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { mdelay(1); msecs--; } ... } Is mdelay(1) a barrier? Yes, because it is a function on x86_64. Absolutely the same code will be buggy on an arch where mdelay(1) == udelay(1000), and udelay is implemented as inline busy-wait. arch/sparc64/kernel/smp.c /* Wait for response */ while (atomic_read(&data.finished) != cpus) cpu_relax(); ...later in the same file... while (atomic_read(&smp_capture_registry) != ncpus) rmb(); I'm confused. Do we need cpu_relax() or rmb()? Does cpu_relax() imply rmb()? (No it doesn't). Which of those two while loops needs correcting? -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 14:16 ` Denys Vlasenko (?) @ 2007-09-10 15:09 ` Linus Torvalds 2007-09-10 16:46 ` Denys Vlasenko ` (2 more replies) -1 siblings, 3 replies; 333+ messages in thread From: Linus Torvalds @ 2007-09-10 15:09 UTC (permalink / raw) To: Denys Vlasenko Cc: Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, 10 Sep 2007, Denys Vlasenko wrote: > > static inline int > qla2x00_wait_for_loop_ready(scsi_qla_host_t *ha) > { > int return_status = QLA_SUCCESS; > unsigned long loop_timeout ; > scsi_qla_host_t *pha = to_qla_parent(ha); > > /* wait for 5 min at the max for loop to be ready */ > loop_timeout = jiffies + (MAX_LOOP_TIMEOUT * HZ); > > while ((!atomic_read(&pha->loop_down_timer) && > atomic_read(&pha->loop_state) == LOOP_DOWN) || > atomic_read(&pha->loop_state) != LOOP_READY) { > if (atomic_read(&pha->loop_state) == LOOP_DEAD) { ... > Is above correct or buggy? Correct, because msleep is a barrier. > Is it obvious? No. It's *buggy*. But it has nothing to do with any msleep() in the loop, or anything else. And more importantly, it would be equally buggy even *with* a "volatile" atomic_read(). Why is this so hard for people to understand? You're all acting like morons. The reason it is buggy has absolutely nothing to do with whether the read is done or not, it has to do with the fact that the CPU may re-order the reads *regardless* of whether the read is done in some specific order by the compiler ot not! In effect, there is zero ordering between all those three reads, and if you don't have memory barriers (or a lock or other serialization), that code is buggy. So stop this idiotic discussion thread already. The above kind of code needs memory barriers to be non-buggy. The whole "volatile or not" discussion is totally idiotic, and pointless, and anybody who doesn't understand that by now needs to just shut up and think about it more, rather than make this discussion drag out even further. The fact is, "volatile" *only* makes things worse. It generates worse code, and never fixes any real bugs. This is a *fact*. Linus ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 15:09 ` Linus Torvalds @ 2007-09-10 16:46 ` Denys Vlasenko 2007-09-10 19:59 ` Kyle Moffett 2007-09-10 18:59 ` Christoph Lameter 2007-09-10 23:19 ` Chris Snook 2 siblings, 1 reply; 333+ messages in thread From: Denys Vlasenko @ 2007-09-10 16:46 UTC (permalink / raw) To: Linus Torvalds Cc: Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Monday 10 September 2007 16:09, Linus Torvalds wrote: > On Mon, 10 Sep 2007, Denys Vlasenko wrote: > > static inline int > > qla2x00_wait_for_loop_ready(scsi_qla_host_t *ha) > > { > > int return_status = QLA_SUCCESS; > > unsigned long loop_timeout ; > > scsi_qla_host_t *pha = to_qla_parent(ha); > > > > /* wait for 5 min at the max for loop to be ready */ > > loop_timeout = jiffies + (MAX_LOOP_TIMEOUT * HZ); > > > > while ((!atomic_read(&pha->loop_down_timer) && > > atomic_read(&pha->loop_state) == LOOP_DOWN) || > > atomic_read(&pha->loop_state) != LOOP_READY) { > > if (atomic_read(&pha->loop_state) == LOOP_DEAD) { > ... > > Is above correct or buggy? Correct, because msleep is a barrier. > > Is it obvious? No. > > It's *buggy*. But it has nothing to do with any msleep() in the loop, or > anything else. > > And more importantly, it would be equally buggy even *with* a "volatile" > atomic_read(). I am not saying that this code is okay, this isn't the point. (The code is in fact awful for several more reasons). My point is that people are confused as to what atomic_read() exactly means, and this is bad. Same for cpu_relax(). First one says "read", and second one doesn't say "barrier". This is real code from current kernel which demonstrates this: "I don't know that cpu_relax() is a barrier already": drivers/kvm/kvm_main.c while (atomic_read(&completed) != needed) { cpu_relax(); barrier(); } "I think that atomic_read() is a read from memory and therefore I don't need a barrier": arch/x86_64/kernel/crash.c msecs = 1000; /* Wait at most a second for the other cpus to stop */ while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) { mdelay(1); msecs--; } Since neither camp seems to give up, I am proposing renaming them to something less confusing, and make everybody happy. cpu_relax_barrier() atomic_value(&x) atomic_fetch(&x) I'm not native English speaker, do these sound better? -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 16:46 ` Denys Vlasenko @ 2007-09-10 19:59 ` Kyle Moffett 0 siblings, 0 replies; 333+ messages in thread From: Kyle Moffett @ 2007-09-10 19:59 UTC (permalink / raw) To: Denys Vlasenko Cc: Linus Torvalds, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Sep 10, 2007, at 12:46:33, Denys Vlasenko wrote: > My point is that people are confused as to what atomic_read() > exactly means, and this is bad. Same for cpu_relax(). First one > says "read", and second one doesn't say "barrier". Q&A: Q: When is it OK to use atomic_read()? A: You are asking the question, so never. Q: But I need to check the value of the atomic at this point in time... A: Your code is buggy if it needs to do that on an atomic_t for anything other than debugging or optimization. Use either atomic_*_return() or a lock and some normal integers. Q: "So why can't the atomic_read DTRT magically?" A: Because "the right thing" depends on the situation and is usually best done with something other than atomic_t. If somebody can post some non-buggy code which is correctly using atomic_read() *and* depends on the compiler generating extra nonsensical loads due to "volatile" then the issue *might* be reconsidered. This also includes samples of code which uses atomic_read() and needs memory barriers (so that we can fix the buggy code, not so we can change atomic_read()). So far the only code samples anybody has posted are buggy regardless of whether or not the value and/or accessors are flagged "volatile" or not. And hey, maybe the volatile ops *should* be implemented in inline ASM for future- proof-ness, but that's a separate issue. Cheers, Kyle Moffett ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-09-10 19:59 ` Kyle Moffett 0 siblings, 0 replies; 333+ messages in thread From: Kyle Moffett @ 2007-09-10 19:59 UTC (permalink / raw) To: Denys Vlasenko Cc: Linus Torvalds, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Sep 10, 2007, at 12:46:33, Denys Vlasenko wrote: > My point is that people are confused as to what atomic_read() > exactly means, and this is bad. Same for cpu_relax(). First one > says "read", and second one doesn't say "barrier". Q&A: Q: When is it OK to use atomic_read()? A: You are asking the question, so never. Q: But I need to check the value of the atomic at this point in time... A: Your code is buggy if it needs to do that on an atomic_t for anything other than debugging or optimization. Use either atomic_*_return() or a lock and some normal integers. Q: "So why can't the atomic_read DTRT magically?" A: Because "the right thing" depends on the situation and is usually best done with something other than atomic_t. If somebody can post some non-buggy code which is correctly using atomic_read() *and* depends on the compiler generating extra nonsensical loads due to "volatile" then the issue *might* be reconsidered. This also includes samples of code which uses atomic_read() and needs memory barriers (so that we can fix the buggy code, not so we can change atomic_read()). So far the only code samples anybody has posted are buggy regardless of whether or not the value and/or accessors are flagged "volatile" or not. And hey, maybe the volatile ops *should* be implemented in inline ASM for future- proof-ness, but that's a separate issue. Cheers, Kyle Moffett ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 15:09 ` Linus Torvalds 2007-09-10 16:46 ` Denys Vlasenko @ 2007-09-10 18:59 ` Christoph Lameter 2007-09-10 23:19 ` Chris Snook 2 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-09-10 18:59 UTC (permalink / raw) To: Linus Torvalds Cc: Denys Vlasenko, Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, 10 Sep 2007, Linus Torvalds wrote: > The fact is, "volatile" *only* makes things worse. It generates worse > code, and never fixes any real bugs. This is a *fact*. Yes, lets just drop the volatiles now! We need a patch that gets rid of them.... Volunteers? ^ permalink raw reply [flat|nested] 333+ messages in thread
* [PATCH] Document non-semantics of atomic_read() and atomic_set() 2007-09-10 15:09 ` Linus Torvalds @ 2007-09-10 23:19 ` Chris Snook 2007-09-10 18:59 ` Christoph Lameter 2007-09-10 23:19 ` Chris Snook 2 siblings, 0 replies; 333+ messages in thread From: Chris Snook @ 2007-09-10 23:19 UTC (permalink / raw) To: Linus Torvalds Cc: Denys Vlasenko, Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher From: Chris Snook <csnook@redhat.com> Unambiguously document the fact that atomic_read() and atomic_set() do not imply any ordering or memory access, and that callers are obligated to explicitly invoke barriers as needed to ensure that changes to atomic variables are visible in all contexts that need to see them. Signed-off-by: Chris Snook <csnook@redhat.com> --- a/Documentation/atomic_ops.txt 2007-07-08 19:32:17.000000000 -0400 +++ b/Documentation/atomic_ops.txt 2007-09-10 19:02:50.000000000 -0400 @@ -12,7 +12,11 @@ C integer type will fail. Something like the following should suffice: - typedef struct { volatile int counter; } atomic_t; + typedef struct { int counter; } atomic_t; + + Historically, counter has been declared volatile. This is now +discouraged. See Documentation/volatile-considered-harmful.txt for the +complete rationale. The first operations to implement for atomic_t's are the initializers and plain reads. @@ -42,6 +46,22 @@ which simply reads the current value of the counter. +*** WARNING: atomic_read() and atomic_set() DO NOT IMPLY BARRIERS! *** + +Some architectures may choose to use the volatile keyword, barriers, or +inline assembly to guarantee some degree of immediacy for atomic_read() +and atomic_set(). This is not uniformly guaranteed, and may change in +the future, so all users of atomic_t should treat atomic_read() and +atomic_set() as simple C assignment statements that may be reordered or +optimized away entirely by the compiler or processor, and explicitly +invoke the appropriate compiler and/or memory barrier for each use case. +Failure to do so will result in code that may suddenly break when used with +different architectures or compiler optimizations, or even changes in +unrelated code which changes how the compiler optimizes the section +accessing atomic_t variables. + +*** YOU HAVE BEEN WARNED! *** + Now, we move onto the actual atomic operation interfaces. void atomic_add(int i, atomic_t *v); ^ permalink raw reply [flat|nested] 333+ messages in thread
* [PATCH] Document non-semantics of atomic_read() and atomic_set() @ 2007-09-10 23:19 ` Chris Snook 0 siblings, 0 replies; 333+ messages in thread From: Chris Snook @ 2007-09-10 23:19 UTC (permalink / raw) To: Linus Torvalds Cc: Denys Vlasenko, Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher From: Chris Snook <csnook@redhat.com> Unambiguously document the fact that atomic_read() and atomic_set() do not imply any ordering or memory access, and that callers are obligated to explicitly invoke barriers as needed to ensure that changes to atomic variables are visible in all contexts that need to see them. Signed-off-by: Chris Snook <csnook@redhat.com> --- a/Documentation/atomic_ops.txt 2007-07-08 19:32:17.000000000 -0400 +++ b/Documentation/atomic_ops.txt 2007-09-10 19:02:50.000000000 -0400 @@ -12,7 +12,11 @@ C integer type will fail. Something like the following should suffice: - typedef struct { volatile int counter; } atomic_t; + typedef struct { int counter; } atomic_t; + + Historically, counter has been declared volatile. This is now +discouraged. See Documentation/volatile-considered-harmful.txt for the +complete rationale. The first operations to implement for atomic_t's are the initializers and plain reads. @@ -42,6 +46,22 @@ which simply reads the current value of the counter. +*** WARNING: atomic_read() and atomic_set() DO NOT IMPLY BARRIERS! *** + +Some architectures may choose to use the volatile keyword, barriers, or +inline assembly to guarantee some degree of immediacy for atomic_read() +and atomic_set(). This is not uniformly guaranteed, and may change in +the future, so all users of atomic_t should treat atomic_read() and +atomic_set() as simple C assignment statements that may be reordered or +optimized away entirely by the compiler or processor, and explicitly +invoke the appropriate compiler and/or memory barrier for each use case. +Failure to do so will result in code that may suddenly break when used with +different architectures or compiler optimizations, or even changes in +unrelated code which changes how the compiler optimizes the section +accessing atomic_t variables. + +*** YOU HAVE BEEN WARNED! *** + Now, we move onto the actual atomic operation interfaces. void atomic_add(int i, atomic_t *v); ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] Document non-semantics of atomic_read() and atomic_set() 2007-09-10 23:19 ` Chris Snook @ 2007-09-10 23:44 ` Paul E. McKenney -1 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-09-10 23:44 UTC (permalink / raw) To: Chris Snook Cc: Linus Torvalds, Denys Vlasenko, Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, Sep 10, 2007 at 07:19:44PM -0400, Chris Snook wrote: > From: Chris Snook <csnook@redhat.com> > > Unambiguously document the fact that atomic_read() and atomic_set() > do not imply any ordering or memory access, and that callers are > obligated to explicitly invoke barriers as needed to ensure that > changes to atomic variables are visible in all contexts that need > to see them. Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> > Signed-off-by: Chris Snook <csnook@redhat.com> > > --- a/Documentation/atomic_ops.txt 2007-07-08 19:32:17.000000000 -0400 > +++ b/Documentation/atomic_ops.txt 2007-09-10 19:02:50.000000000 -0400 > @@ -12,7 +12,11 @@ > C integer type will fail. Something like the following should > suffice: > > - typedef struct { volatile int counter; } atomic_t; > + typedef struct { int counter; } atomic_t; > + > + Historically, counter has been declared volatile. This is now > +discouraged. See Documentation/volatile-considered-harmful.txt for the > +complete rationale. > > The first operations to implement for atomic_t's are the > initializers and plain reads. > @@ -42,6 +46,22 @@ > > which simply reads the current value of the counter. > > +*** WARNING: atomic_read() and atomic_set() DO NOT IMPLY BARRIERS! *** > + > +Some architectures may choose to use the volatile keyword, barriers, or > +inline assembly to guarantee some degree of immediacy for atomic_read() > +and atomic_set(). This is not uniformly guaranteed, and may change in > +the future, so all users of atomic_t should treat atomic_read() and > +atomic_set() as simple C assignment statements that may be reordered or > +optimized away entirely by the compiler or processor, and explicitly > +invoke the appropriate compiler and/or memory barrier for each use case. > +Failure to do so will result in code that may suddenly break when used with > +different architectures or compiler optimizations, or even changes in > +unrelated code which changes how the compiler optimizes the section > +accessing atomic_t variables. > + > +*** YOU HAVE BEEN WARNED! *** > + > Now, we move onto the actual atomic operation interfaces. > > void atomic_add(int i, atomic_t *v); ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] Document non-semantics of atomic_read() and atomic_set() @ 2007-09-10 23:44 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-09-10 23:44 UTC (permalink / raw) To: Chris Snook Cc: Linus Torvalds, Denys Vlasenko, Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Ilpo Jarvinen, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, Sep 10, 2007 at 07:19:44PM -0400, Chris Snook wrote: > From: Chris Snook <csnook@redhat.com> > > Unambiguously document the fact that atomic_read() and atomic_set() > do not imply any ordering or memory access, and that callers are > obligated to explicitly invoke barriers as needed to ensure that > changes to atomic variables are visible in all contexts that need > to see them. Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> > Signed-off-by: Chris Snook <csnook@redhat.com> > > --- a/Documentation/atomic_ops.txt 2007-07-08 19:32:17.000000000 -0400 > +++ b/Documentation/atomic_ops.txt 2007-09-10 19:02:50.000000000 -0400 > @@ -12,7 +12,11 @@ > C integer type will fail. Something like the following should > suffice: > > - typedef struct { volatile int counter; } atomic_t; > + typedef struct { int counter; } atomic_t; > + > + Historically, counter has been declared volatile. This is now > +discouraged. See Documentation/volatile-considered-harmful.txt for the > +complete rationale. > > The first operations to implement for atomic_t's are the > initializers and plain reads. > @@ -42,6 +46,22 @@ > > which simply reads the current value of the counter. > > +*** WARNING: atomic_read() and atomic_set() DO NOT IMPLY BARRIERS! *** > + > +Some architectures may choose to use the volatile keyword, barriers, or > +inline assembly to guarantee some degree of immediacy for atomic_read() > +and atomic_set(). This is not uniformly guaranteed, and may change in > +the future, so all users of atomic_t should treat atomic_read() and > +atomic_set() as simple C assignment statements that may be reordered or > +optimized away entirely by the compiler or processor, and explicitly > +invoke the appropriate compiler and/or memory barrier for each use case. > +Failure to do so will result in code that may suddenly break when used with > +different architectures or compiler optimizations, or even changes in > +unrelated code which changes how the compiler optimizes the section > +accessing atomic_t variables. > + > +*** YOU HAVE BEEN WARNED! *** > + > Now, we move onto the actual atomic operation interfaces. > > void atomic_add(int i, atomic_t *v); ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] Document non-semantics of atomic_read() and atomic_set() 2007-09-10 23:19 ` Chris Snook @ 2007-09-11 19:35 ` Christoph Lameter -1 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-09-11 19:35 UTC (permalink / raw) To: Chris Snook Cc: Linus Torvalds, Denys Vlasenko, Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Acked-by: Christoph Lameter <clameter@sgi.com> ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH] Document non-semantics of atomic_read() and atomic_set() @ 2007-09-11 19:35 ` Christoph Lameter 0 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-09-11 19:35 UTC (permalink / raw) To: Chris Snook Cc: Linus Torvalds, Denys Vlasenko, Kyle Moffett, Arjan van de Ven, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Acked-by: Christoph Lameter <clameter@sgi.com> ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 10:56 ` Denys Vlasenko 2007-09-10 11:15 ` Herbert Xu 2007-09-10 12:22 ` Kyle Moffett @ 2007-09-10 14:51 ` Arjan van de Ven 2007-09-10 14:38 ` Denys Vlasenko 2 siblings, 1 reply; 333+ messages in thread From: Arjan van de Ven @ 2007-09-10 14:51 UTC (permalink / raw) To: Denys Vlasenko Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, 10 Sep 2007 11:56:29 +0100 Denys Vlasenko <vda.linux@googlemail.com> wrote: > > Well, if you insist on having it again: > > Waiting for atomic value to be zero: > > while (atomic_read(&x)) > continue; > and this I would say is buggy code all the way. Not from a pure C level semantics, but from a "busy waiting is buggy" semantics level and a "I'm inventing my own locking" semantics level. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 14:51 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Arjan van de Ven @ 2007-09-10 14:38 ` Denys Vlasenko 2007-09-10 17:02 ` Arjan van de Ven 0 siblings, 1 reply; 333+ messages in thread From: Denys Vlasenko @ 2007-09-10 14:38 UTC (permalink / raw) To: Arjan van de Ven Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Monday 10 September 2007 15:51, Arjan van de Ven wrote: > On Mon, 10 Sep 2007 11:56:29 +0100 > Denys Vlasenko <vda.linux@googlemail.com> wrote: > > > > > Well, if you insist on having it again: > > > > Waiting for atomic value to be zero: > > > > while (atomic_read(&x)) > > continue; > > > > and this I would say is buggy code all the way. > > Not from a pure C level semantics, but from a "busy waiting is buggy" > semantics level and a "I'm inventing my own locking" semantics level. After inspecting arch/*, I cannot agree with you. Otherwise almost all major architectures use "conceptually buggy busy-waiting": arch/alpha arch/i386 arch/ia64 arch/m32r arch/mips arch/parisc arch/powerpc arch/sh arch/sparc64 arch/um arch/x86_64 All of the above contain busy-waiting on atomic_read. Including these loops without barriers: arch/mips/kernel/smtc.c while (atomic_read(&idle_hook_initialized) < 1000) ; arch/mips/sgi-ip27/ip27-nmi.c while (atomic_read(&nmied_cpus) != num_online_cpus()); [Well maybe num_online_cpus() is a barrier, I didn't check] arch/sh/kernel/smp.c if (wait) while (atomic_read(&smp_fn_call.finished) != (nr_cpus - 1)); Bugs? -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 14:38 ` Denys Vlasenko @ 2007-09-10 17:02 ` Arjan van de Ven 0 siblings, 0 replies; 333+ messages in thread From: Arjan van de Ven @ 2007-09-10 17:02 UTC (permalink / raw) To: Denys Vlasenko Cc: Linus Torvalds, Nick Piggin, Satyam Sharma, Herbert Xu, Paul Mackerras, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Mon, 10 Sep 2007 15:38:23 +0100 Denys Vlasenko <vda.linux@googlemail.com> wrote: > On Monday 10 September 2007 15:51, Arjan van de Ven wrote: > > On Mon, 10 Sep 2007 11:56:29 +0100 > > Denys Vlasenko <vda.linux@googlemail.com> wrote: > > > > > > > > Well, if you insist on having it again: > > > > > > Waiting for atomic value to be zero: > > > > > > while (atomic_read(&x)) > > > continue; > > > > > > > and this I would say is buggy code all the way. > > > > Not from a pure C level semantics, but from a "busy waiting is > > buggy" semantics level and a "I'm inventing my own locking" > > semantics level. > > After inspecting arch/*, I cannot agree with you. the arch/ people obviously are allowed to do their own locking stuff... BECAUSE THEY HAVE TO IMPLEMENT THAT! the arch maintainers know EXACTLY how their hw behaves (well, we hope) so they tend to be the exception to many rules in the kernel.... ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 8:38 ` Nick Piggin 2007-08-17 9:14 ` Satyam Sharma @ 2007-08-17 11:08 ` Stefan Richter 1 sibling, 0 replies; 333+ messages in thread From: Stefan Richter @ 2007-08-17 11:08 UTC (permalink / raw) To: Nick Piggin Cc: Satyam Sharma, Herbert Xu, Paul Mackerras, Linus Torvalds, Christoph Lameter, Chris Snook, Ilpo Jarvinen, Paul E. McKenney, Linux Kernel Mailing List, linux-arch, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Nick Piggin wrote: > Satyam Sharma wrote: >> And we have driver / subsystem maintainers such as Stefan >> coming up and admitting that often a lot of code that's written to use >> atomic_read() does assume the read will not be elided by the compiler. > > So these are broken on i386 and x86-64? The ieee1394 and firewire subsystems have open, undiagnosed bugs, also on i386 and x86-64. But whether there is any bug because of wrong assumptions about atomic_read among them, I don't know. I don't know which assumptions the authors made, I only know that I wasn't aware of all the properties of atomic_read until now. > Are they definitely safe on SMP and weakly ordered machines with > just a simple compiler barrier there? Because I would not be > surprised if there are a lot of developers who don't really know > what to assume when it comes to memory ordering issues. > > This is not a dig at driver writers: we still have memory ordering > problems in the VM too (and probably most of the subtle bugs in > lockless VM code are memory ordering ones). Let's not make up a > false sense of security and hope that sprinkling volatile around > will allow people to write bug-free lockless code. If a writer > can't be bothered reading API documentation ...or, if there is none, the implementation specification (as in case of the atomic ops), or, if there is none, the implementation (as in case of a some infrastructure code here and there)... > and learning the Linux memory model, they can still be productive > writing safely locked code. Provided they are aware that they might not have the full picture of the lockless primitives. :-) -- Stefan Richter -=====-=-=== =--- =---= http://arcgraph.de/sr/ ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 3:03 ` Linus Torvalds @ 2007-08-17 22:09 ` Segher Boessenkool 2007-08-17 22:09 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 22:09 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Ilpo J?rvinen, Linux Kernel Mailing List, David Miller, Paul E. McKenney, ak, Netdev, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, wensong, wjiang > Of course, since *normal* accesses aren't necessarily limited wrt > re-ordering, the question then becomes one of "with regard to *what* > does > it limit re-ordering?". > > A C compiler that re-orders two different volatile accesses that have a > sequence point in between them is pretty clearly a buggy compiler. So > at a > minimum, it limits re-ordering wrt other volatiles (assuming sequence > points exists). It also means that the compiler cannot move it > speculatively across conditionals, but other than that it's starting to > get fuzzy. This is actually really well-defined in C, not fuzzy at all. "Volatile accesses" are a side effect, and no side effects can be reordered with respect to sequence points. The side effects that matter in the kernel environment are: 1) accessing a volatile object; 2) modifying an object; 3) volatile asm(); 4) calling a function that does any of these. We certainly should avoid volatile whenever possible, but "because it's fuzzy wrt reordering" is not a reason -- all alternatives have exactly the same issues. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-17 22:09 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 22:09 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Ilpo J?rvinen, Linux Kernel Mailing List, David Miller, Paul E. McKenney, ak, Netdev, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, wensong, wjiang > Of course, since *normal* accesses aren't necessarily limited wrt > re-ordering, the question then becomes one of "with regard to *what* > does > it limit re-ordering?". > > A C compiler that re-orders two different volatile accesses that have a > sequence point in between them is pretty clearly a buggy compiler. So > at a > minimum, it limits re-ordering wrt other volatiles (assuming sequence > points exists). It also means that the compiler cannot move it > speculatively across conditionals, but other than that it's starting to > get fuzzy. This is actually really well-defined in C, not fuzzy at all. "Volatile accesses" are a side effect, and no side effects can be reordered with respect to sequence points. The side effects that matter in the kernel environment are: 1) accessing a volatile object; 2) modifying an object; 3) volatile asm(); 4) calling a function that does any of these. We certainly should avoid volatile whenever possible, but "because it's fuzzy wrt reordering" is not a reason -- all alternatives have exactly the same issues. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 20:20 ` Christoph Lameter 2007-08-17 1:02 ` Paul E. McKenney 2007-08-17 2:16 ` Paul Mackerras @ 2007-08-17 17:41 ` Segher Boessenkool 2007-08-17 18:38 ` Satyam Sharma 2007-09-10 18:59 ` Christoph Lameter 2 siblings, 2 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 17:41 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >> atomic_dec() already has volatile behavior everywhere, so this is >> semantically >> okay, but this code (and any like it) should be calling cpu_relax() >> each >> iteration through the loop, unless there's a compelling reason not >> to. I'll >> allow that for some hardware drivers (possibly this one) such a >> compelling >> reason may exist, but hardware-independent core subsystems probably >> have no >> excuse. > > No it does not have any volatile semantics. atomic_dec() can be > reordered > at will by the compiler within the current basic unit if you do not > add a > barrier. "volatile" has nothing to do with reordering. atomic_dec() writes to memory, so it _does_ have "volatile semantics", implicitly, as long as the compiler cannot optimise the atomic variable away completely -- any store counts as a side effect. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 17:41 ` Segher Boessenkool @ 2007-08-17 18:38 ` Satyam Sharma 2007-08-17 23:17 ` Segher Boessenkool 2007-09-10 18:59 ` Christoph Lameter 1 sibling, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 18:38 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Fri, 17 Aug 2007, Segher Boessenkool wrote: > > > atomic_dec() already has volatile behavior everywhere, so this is > > > semantically > > > okay, but this code (and any like it) should be calling cpu_relax() each > > > iteration through the loop, unless there's a compelling reason not to. > > > I'll > > > allow that for some hardware drivers (possibly this one) such a compelling > > > reason may exist, but hardware-independent core subsystems probably have > > > no > > > excuse. > > > > No it does not have any volatile semantics. atomic_dec() can be reordered > > at will by the compiler within the current basic unit if you do not add a > > barrier. > > "volatile" has nothing to do with reordering. If you're talking of "volatile" the type-qualifier keyword, then http://lkml.org/lkml/2007/8/16/231 (and sub-thread below it) shows otherwise. > atomic_dec() writes > to memory, so it _does_ have "volatile semantics", implicitly, as > long as the compiler cannot optimise the atomic variable away > completely -- any store counts as a side effect. I don't think an atomic_dec() implemented as an inline "asm volatile" or one that uses a "forget" macro would have the same re-ordering guarantees as an atomic_dec() that uses a volatile access cast. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 18:38 ` Satyam Sharma @ 2007-08-17 23:17 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 23:17 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >>> No it does not have any volatile semantics. atomic_dec() can be >>> reordered >>> at will by the compiler within the current basic unit if you do not >>> add a >>> barrier. >> >> "volatile" has nothing to do with reordering. > > If you're talking of "volatile" the type-qualifier keyword, then > http://lkml.org/lkml/2007/8/16/231 (and sub-thread below it) shows > otherwise. I'm not sure what in that mail you mean, but anyway... Yes, of course, the fact that "volatile" creates a side effect prevents certain things from being reordered wrt the atomic_dec(); but the atomic_dec() has a side effect *already* so the volatile doesn't change anything. >> atomic_dec() writes >> to memory, so it _does_ have "volatile semantics", implicitly, as >> long as the compiler cannot optimise the atomic variable away >> completely -- any store counts as a side effect. > > I don't think an atomic_dec() implemented as an inline "asm volatile" > or one that uses a "forget" macro would have the same re-ordering > guarantees as an atomic_dec() that uses a volatile access cast. The "asm volatile" implementation does have exactly the same reordering guarantees as the "volatile cast" thing, if that is implemented by GCC in the "obvious" way. Even a "plain" asm() will do the same. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-17 23:17 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 23:17 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >>> No it does not have any volatile semantics. atomic_dec() can be >>> reordered >>> at will by the compiler within the current basic unit if you do not >>> add a >>> barrier. >> >> "volatile" has nothing to do with reordering. > > If you're talking of "volatile" the type-qualifier keyword, then > http://lkml.org/lkml/2007/8/16/231 (and sub-thread below it) shows > otherwise. I'm not sure what in that mail you mean, but anyway... Yes, of course, the fact that "volatile" creates a side effect prevents certain things from being reordered wrt the atomic_dec(); but the atomic_dec() has a side effect *already* so the volatile doesn't change anything. >> atomic_dec() writes >> to memory, so it _does_ have "volatile semantics", implicitly, as >> long as the compiler cannot optimise the atomic variable away >> completely -- any store counts as a side effect. > > I don't think an atomic_dec() implemented as an inline "asm volatile" > or one that uses a "forget" macro would have the same re-ordering > guarantees as an atomic_dec() that uses a volatile access cast. The "asm volatile" implementation does have exactly the same reordering guarantees as the "volatile cast" thing, if that is implemented by GCC in the "obvious" way. Even a "plain" asm() will do the same. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 23:17 ` Segher Boessenkool (?) @ 2007-08-17 23:55 ` Satyam Sharma 2007-08-18 0:04 ` Segher Boessenkool -1 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 23:55 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Sat, 18 Aug 2007, Segher Boessenkool wrote: > > > > No it does not have any volatile semantics. atomic_dec() can be > > > > reordered > > > > at will by the compiler within the current basic unit if you do not add > > > > a > > > > barrier. > > > > > > "volatile" has nothing to do with reordering. > > > > If you're talking of "volatile" the type-qualifier keyword, then > > http://lkml.org/lkml/2007/8/16/231 (and sub-thread below it) shows > > otherwise. > > I'm not sure what in that mail you mean, but anyway... > > Yes, of course, the fact that "volatile" creates a side effect > prevents certain things from being reordered wrt the atomic_dec(); > but the atomic_dec() has a side effect *already* so the volatile > doesn't change anything. That's precisely what that sub-thread (read down to the last mail there, and not the first mail only) shows. So yes, "volatile" does have something to do with re-ordering (as guaranteed by the C standard). > > > atomic_dec() writes > > > to memory, so it _does_ have "volatile semantics", implicitly, as > > > long as the compiler cannot optimise the atomic variable away > > > completely -- any store counts as a side effect. > > > > I don't think an atomic_dec() implemented as an inline "asm volatile" > > or one that uses a "forget" macro would have the same re-ordering > > guarantees as an atomic_dec() that uses a volatile access cast. > > The "asm volatile" implementation does have exactly the same > reordering guarantees as the "volatile cast" thing, I don't think so. > if that is > implemented by GCC in the "obvious" way. Even a "plain" asm() > will do the same. Read the relevant GCC documentation. [ of course, if the (latest) GCC documentation is *yet again* wrong, then alright, not much I can do about it, is there. ] ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 23:55 ` Satyam Sharma @ 2007-08-18 0:04 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-18 0:04 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >>>> atomic_dec() writes >>>> to memory, so it _does_ have "volatile semantics", implicitly, as >>>> long as the compiler cannot optimise the atomic variable away >>>> completely -- any store counts as a side effect. >>> >>> I don't think an atomic_dec() implemented as an inline "asm volatile" >>> or one that uses a "forget" macro would have the same re-ordering >>> guarantees as an atomic_dec() that uses a volatile access cast. >> >> The "asm volatile" implementation does have exactly the same >> reordering guarantees as the "volatile cast" thing, > > I don't think so. "asm volatile" creates a side effect. Side effects aren't allowed to be reordered wrt sequence points. This is exactly the same reason as why "volatile accesses" cannot be reordered. >> if that is >> implemented by GCC in the "obvious" way. Even a "plain" asm() >> will do the same. > > Read the relevant GCC documentation. I did, yes. > [ of course, if the (latest) GCC documentation is *yet again* > wrong, then alright, not much I can do about it, is there. ] There was (and is) nothing wrong about the "+m" documentation, if that is what you are talking about. It could be extended now, to allow "+m" -- but that takes more than just "fixing" the documentation. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-18 0:04 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-18 0:04 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >>>> atomic_dec() writes >>>> to memory, so it _does_ have "volatile semantics", implicitly, as >>>> long as the compiler cannot optimise the atomic variable away >>>> completely -- any store counts as a side effect. >>> >>> I don't think an atomic_dec() implemented as an inline "asm volatile" >>> or one that uses a "forget" macro would have the same re-ordering >>> guarantees as an atomic_dec() that uses a volatile access cast. >> >> The "asm volatile" implementation does have exactly the same >> reordering guarantees as the "volatile cast" thing, > > I don't think so. "asm volatile" creates a side effect. Side effects aren't allowed to be reordered wrt sequence points. This is exactly the same reason as why "volatile accesses" cannot be reordered. >> if that is >> implemented by GCC in the "obvious" way. Even a "plain" asm() >> will do the same. > > Read the relevant GCC documentation. I did, yes. > [ of course, if the (latest) GCC documentation is *yet again* > wrong, then alright, not much I can do about it, is there. ] There was (and is) nothing wrong about the "+m" documentation, if that is what you are talking about. It could be extended now, to allow "+m" -- but that takes more than just "fixing" the documentation. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 0:04 ` Segher Boessenkool (?) @ 2007-08-18 1:56 ` Satyam Sharma 2007-08-18 2:15 ` Segher Boessenkool -1 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-18 1:56 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Sat, 18 Aug 2007, Segher Boessenkool wrote: > > > > > atomic_dec() writes > > > > > to memory, so it _does_ have "volatile semantics", implicitly, as > > > > > long as the compiler cannot optimise the atomic variable away > > > > > completely -- any store counts as a side effect. > > > > > > > > I don't think an atomic_dec() implemented as an inline "asm volatile" > > > > or one that uses a "forget" macro would have the same re-ordering > > > > guarantees as an atomic_dec() that uses a volatile access cast. > > > > > > The "asm volatile" implementation does have exactly the same > > > reordering guarantees as the "volatile cast" thing, > > > > I don't think so. > > "asm volatile" creates a side effect. Yeah. > Side effects aren't > allowed to be reordered wrt sequence points. Yeah. > This is exactly > the same reason as why "volatile accesses" cannot be reordered. No, the code in that sub-thread I earlier pointed you at *WAS* written such that there was a sequence point after all the uses of that volatile access cast macro, and _therefore_ we were safe from re-ordering (behaviour guaranteed by the C standard). But you seem to be missing the simple and basic fact that: (something_that_has_side_effects || statement) != something_that_is_a_sequence_point Now, one cannot fantasize that "volatile asms" are also sequence points. In fact such an argument would be sadly mistaken, because "sequence points" are defined by the C standard and it'd be horribly wrong to even _try_ claiming that the C standard knows about "volatile asms". > > > if that is > > > implemented by GCC in the "obvious" way. Even a "plain" asm() > > > will do the same. > > > > Read the relevant GCC documentation. > > I did, yes. No, you didn't read: http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html Read the bit about the need for artificial dependencies, and the example given there: asm volatile("mtfsf 255,%0" : : "f" (fpenv)); sum = x + y; The docs explicitly say the addition can be moved before the "volatile asm". Hopefully, as you know, (x + y) is an C "expression" and hence a "sequence point" as defined by the standard. So the "volatile asm" should've happened before it, right? Wrong. I know there is also stuff written about "side-effects" there which _could_ give the same semantic w.r.t. sequence points as the volatile access casts, but hey, it's GCC's own documentation, you obviously can't find fault with _me_ if there's wrong stuff written in there. Say that to GCC ... See, "volatile" C keyword, for all it's ill-definition and dodgy semantics, is still at least given somewhat of a treatment in the C standard (whose quality is ... ummm, sadly not always good and clear, but unsurprisingly, still about 5,482 orders-of-magnitude times better than GCC docs). Semantics of "volatile" as applies to inline asm, OTOH? You're completely relying on the compiler for that ... > > [ of course, if the (latest) GCC documentation is *yet again* > > wrong, then alright, not much I can do about it, is there. ] > > There was (and is) nothing wrong about the "+m" documentation, if > that is what you are talking about. It could be extended now, to > allow "+m" -- but that takes more than just "fixing" the documentation. No, there was (and is) _everything_ wrong about the "+" documentation as applies to memory-constrained operands. I don't give a whit if it's some workaround in their gimplifier, or the other, that makes it possible to use "+m" (like the current kernel code does). The docs suggest otherwise, so there's obviously a clear disconnect between the docs and actual GCC behaviour. [ You seem to often take issue with _amazingly_ petty and pedantic things, by the way :-) ] ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 1:56 ` Satyam Sharma @ 2007-08-18 2:15 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-18 2:15 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >>>> The "asm volatile" implementation does have exactly the same >>>> reordering guarantees as the "volatile cast" thing, >>> >>> I don't think so. >> >> "asm volatile" creates a side effect. > > Yeah. > >> Side effects aren't >> allowed to be reordered wrt sequence points. > > Yeah. > >> This is exactly >> the same reason as why "volatile accesses" cannot be reordered. > > No, the code in that sub-thread I earlier pointed you at *WAS* written > such that there was a sequence point after all the uses of that > volatile > access cast macro, and _therefore_ we were safe from re-ordering > (behaviour guaranteed by the C standard). And exactly the same is true for the "asm" version. > Now, one cannot fantasize that "volatile asms" are also sequence > points. Sure you can do that. I don't though. > In fact such an argument would be sadly mistaken, because "sequence > points" are defined by the C standard and it'd be horribly wrong to > even _try_ claiming that the C standard knows about "volatile asms". That's nonsense. GCC can extend the C standard any way they bloody well please -- witness the fact that they added an extra class of side effects... >>> Read the relevant GCC documentation. >> >> I did, yes. > > No, you didn't read: > > http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html > > Read the bit about the need for artificial dependencies, and the > example > given there: > > asm volatile("mtfsf 255,%0" : : "f" (fpenv)); > sum = x + y; > > The docs explicitly say the addition can be moved before the "volatile > asm". Hopefully, as you know, (x + y) is an C "expression" and hence > a "sequence point" as defined by the standard. The _end of a full expression_ is a sequence point, not every expression. And that is irrelevant here anyway. It is perfectly fine to compute x+y any time before the assignment; the C compiler is allowed to compute it _after_ the assignment even, if it could figure out how ;-) x+y does not contain a side effect, you know. > I know there is also stuff written about "side-effects" there which > _could_ give the same semantic w.r.t. sequence points as the volatile > access casts, s/could/does/ > but hey, it's GCC's own documentation, you obviously can't > find fault with _me_ if there's wrong stuff written in there. Say that > to GCC ... There's nothing wrong there. > See, "volatile" C keyword, for all it's ill-definition and dodgy > semantics, is still at least given somewhat of a treatment in the C > standard (whose quality is ... ummm, sadly not always good and clear, > but unsurprisingly, still about 5,482 orders-of-magnitude times > better than GCC docs). If you find any problems/shortcomings in the GCC documentation, please file a PR, don't go whine on some unrelated mailing lists. Thank you. > Semantics of "volatile" as applies to inline > asm, OTOH? You're completely relying on the compiler for that ... Yes, and? GCC promises the behaviour it has documented. >>> [ of course, if the (latest) GCC documentation is *yet again* >>> wrong, then alright, not much I can do about it, is there. ] >> >> There was (and is) nothing wrong about the "+m" documentation, if >> that is what you are talking about. It could be extended now, to >> allow "+m" -- but that takes more than just "fixing" the >> documentation. > > No, there was (and is) _everything_ wrong about the "+" documentation > as > applies to memory-constrained operands. I don't give a whit if it's > some workaround in their gimplifier, or the other, that makes it > possible > to use "+m" (like the current kernel code does). The docs suggest > otherwise, so there's obviously a clear disconnect between the docs and > actual GCC behaviour. The documentation simply doesn't say "+m" is allowed. The code to allow it was added for the benefit of people who do not read the documentation. Documentation for "+m" might get added later if it is decided this [the code, not the documentation] is a sane thing to have (which isn't directly obvious). > [ You seem to often take issue with _amazingly_ petty and pedantic > things, > by the way :-) ] If you're talking details, you better get them right. Handwaving is fine with me as long as you're not purporting you're not. And I simply cannot stand false assertions. You can always ignore me if _you_ take issue with _that_ :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-18 2:15 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-18 2:15 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >>>> The "asm volatile" implementation does have exactly the same >>>> reordering guarantees as the "volatile cast" thing, >>> >>> I don't think so. >> >> "asm volatile" creates a side effect. > > Yeah. > >> Side effects aren't >> allowed to be reordered wrt sequence points. > > Yeah. > >> This is exactly >> the same reason as why "volatile accesses" cannot be reordered. > > No, the code in that sub-thread I earlier pointed you at *WAS* written > such that there was a sequence point after all the uses of that > volatile > access cast macro, and _therefore_ we were safe from re-ordering > (behaviour guaranteed by the C standard). And exactly the same is true for the "asm" version. > Now, one cannot fantasize that "volatile asms" are also sequence > points. Sure you can do that. I don't though. > In fact such an argument would be sadly mistaken, because "sequence > points" are defined by the C standard and it'd be horribly wrong to > even _try_ claiming that the C standard knows about "volatile asms". That's nonsense. GCC can extend the C standard any way they bloody well please -- witness the fact that they added an extra class of side effects... >>> Read the relevant GCC documentation. >> >> I did, yes. > > No, you didn't read: > > http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html > > Read the bit about the need for artificial dependencies, and the > example > given there: > > asm volatile("mtfsf 255,%0" : : "f" (fpenv)); > sum = x + y; > > The docs explicitly say the addition can be moved before the "volatile > asm". Hopefully, as you know, (x + y) is an C "expression" and hence > a "sequence point" as defined by the standard. The _end of a full expression_ is a sequence point, not every expression. And that is irrelevant here anyway. It is perfectly fine to compute x+y any time before the assignment; the C compiler is allowed to compute it _after_ the assignment even, if it could figure out how ;-) x+y does not contain a side effect, you know. > I know there is also stuff written about "side-effects" there which > _could_ give the same semantic w.r.t. sequence points as the volatile > access casts, s/could/does/ > but hey, it's GCC's own documentation, you obviously can't > find fault with _me_ if there's wrong stuff written in there. Say that > to GCC ... There's nothing wrong there. > See, "volatile" C keyword, for all it's ill-definition and dodgy > semantics, is still at least given somewhat of a treatment in the C > standard (whose quality is ... ummm, sadly not always good and clear, > but unsurprisingly, still about 5,482 orders-of-magnitude times > better than GCC docs). If you find any problems/shortcomings in the GCC documentation, please file a PR, don't go whine on some unrelated mailing lists. Thank you. > Semantics of "volatile" as applies to inline > asm, OTOH? You're completely relying on the compiler for that ... Yes, and? GCC promises the behaviour it has documented. >>> [ of course, if the (latest) GCC documentation is *yet again* >>> wrong, then alright, not much I can do about it, is there. ] >> >> There was (and is) nothing wrong about the "+m" documentation, if >> that is what you are talking about. It could be extended now, to >> allow "+m" -- but that takes more than just "fixing" the >> documentation. > > No, there was (and is) _everything_ wrong about the "+" documentation > as > applies to memory-constrained operands. I don't give a whit if it's > some workaround in their gimplifier, or the other, that makes it > possible > to use "+m" (like the current kernel code does). The docs suggest > otherwise, so there's obviously a clear disconnect between the docs and > actual GCC behaviour. The documentation simply doesn't say "+m" is allowed. The code to allow it was added for the benefit of people who do not read the documentation. Documentation for "+m" might get added later if it is decided this [the code, not the documentation] is a sane thing to have (which isn't directly obvious). > [ You seem to often take issue with _amazingly_ petty and pedantic > things, > by the way :-) ] If you're talking details, you better get them right. Handwaving is fine with me as long as you're not purporting you're not. And I simply cannot stand false assertions. You can always ignore me if _you_ take issue with _that_ :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 2:15 ` Segher Boessenkool (?) @ 2007-08-18 3:33 ` Satyam Sharma 2007-08-18 5:18 ` Segher Boessenkool -1 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-18 3:33 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Sat, 18 Aug 2007, Segher Boessenkool wrote: > > > > > The "asm volatile" implementation does have exactly the same > > > > > reordering guarantees as the "volatile cast" thing, > > > > > > > > I don't think so. > > > > > > "asm volatile" creates a side effect. > > > > Yeah. > > > > > Side effects aren't > > > allowed to be reordered wrt sequence points. > > > > Yeah. > > > > > This is exactly > > > the same reason as why "volatile accesses" cannot be reordered. > > > > No, the code in that sub-thread I earlier pointed you at *WAS* written > > such that there was a sequence point after all the uses of that volatile > > access cast macro, and _therefore_ we were safe from re-ordering > > (behaviour guaranteed by the C standard). > > And exactly the same is true for the "asm" version. > > > Now, one cannot fantasize that "volatile asms" are also sequence points. > > Sure you can do that. I don't though. > > > In fact such an argument would be sadly mistaken, because "sequence > > points" are defined by the C standard and it'd be horribly wrong to > > even _try_ claiming that the C standard knows about "volatile asms". > > That's nonsense. GCC can extend the C standard any way they > bloody well please -- witness the fact that they added an > extra class of side effects... > > > > > Read the relevant GCC documentation. > > > > > > I did, yes. > > > > No, you didn't read: > > > > http://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html > > > > Read the bit about the need for artificial dependencies, and the example > > given there: > > > > asm volatile("mtfsf 255,%0" : : "f" (fpenv)); > > sum = x + y; > > > > The docs explicitly say the addition can be moved before the "volatile > > asm". Hopefully, as you know, (x + y) is an C "expression" and hence > > a "sequence point" as defined by the standard. > > The _end of a full expression_ is a sequence point, not every > expression. And that is irrelevant here anyway. > > It is perfectly fine to compute x+y any time before the > assignment; the C compiler is allowed to compute it _after_ > the assignment even, if it could figure out how ;-) > > x+y does not contain a side effect, you know. > > > I know there is also stuff written about "side-effects" there which > > _could_ give the same semantic w.r.t. sequence points as the volatile > > access casts, > > s/could/does/ > > > but hey, it's GCC's own documentation, you obviously can't > > find fault with _me_ if there's wrong stuff written in there. Say that > > to GCC ... > > There's nothing wrong there. > > > See, "volatile" C keyword, for all it's ill-definition and dodgy > > semantics, is still at least given somewhat of a treatment in the C > > standard (whose quality is ... ummm, sadly not always good and clear, > > but unsurprisingly, still about 5,482 orders-of-magnitude times > > better than GCC docs). > > If you find any problems/shortcomings in the GCC documentation, > please file a PR, don't go whine on some unrelated mailing lists. > Thank you. > > > Semantics of "volatile" as applies to inline > > asm, OTOH? You're completely relying on the compiler for that ... > > Yes, and? GCC promises the behaviour it has documented. LOTS there, which obviously isn't correct, but which I'll reply to later, easier stuff first. (call this "handwaving" if you want, but don't worry, I /will/ bother myself to reply) > > > > [ of course, if the (latest) GCC documentation is *yet again* > > > > wrong, then alright, not much I can do about it, is there. ] > > > > > > There was (and is) nothing wrong about the "+m" documentation, if > > > that is what you are talking about. It could be extended now, to > > > allow "+m" -- but that takes more than just "fixing" the documentation. > > > > No, there was (and is) _everything_ wrong about the "+" documentation as > > applies to memory-constrained operands. I don't give a whit if it's > > some workaround in their gimplifier, or the other, that makes it possible > > to use "+m" (like the current kernel code does). The docs suggest > > otherwise, so there's obviously a clear disconnect between the docs and > > actual GCC behaviour. > > The documentation simply doesn't say "+m" is allowed. The code to > allow it was added for the benefit of people who do not read the > documentation. Documentation for "+m" might get added later if it > is decided this [the code, not the documentation] is a sane thing > to have (which isn't directly obvious). Huh? "If the (current) documentation doesn't match up with the (current) code, then _at least one_ of them has to be (as of current) wrong." I wonder how could you even try to disagree with that. And I didn't go whining about this ... you asked me. (I think I'd said something to the effect of GCC docs are often wrong, which is true, but probably you feel saying that is "not allowed" on non-gcc lists?) As for the "PR" you're requesting me to file with GCC for this, that gcc-patches@ thread did precisely that and more (submitted a patch to said documentation -- and no, saying "documentation might get added later" is totally bogus and nonsensical -- documentation exists to document current behaviour, not past). But come on, this is wholly petty. I wouldn't have replied, really, if you weren't so provoking. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 3:33 ` Satyam Sharma @ 2007-08-18 5:18 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-18 5:18 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >> The documentation simply doesn't say "+m" is allowed. The code to >> allow it was added for the benefit of people who do not read the >> documentation. Documentation for "+m" might get added later if it >> is decided this [the code, not the documentation] is a sane thing >> to have (which isn't directly obvious). > > Huh? > > "If the (current) documentation doesn't match up with the (current) > code, then _at least one_ of them has to be (as of current) wrong." > > I wonder how could you even try to disagree with that. Easy. The GCC documentation you're referring to is the user's manual. See the blurb on the first page: "This manual documents how to use the GNU compilers, as well as their features and incompatibilities, and how to report bugs. It corresponds to GCC version 4.3.0. The internals of the GNU compilers, including how to port them to new targets and some information about how to write front ends for new languages, are documented in a separate manual." _How to use_. This documentation doesn't describe in minute detail everything the compiler does (see the source code for that -- no, it isn't described in the internals manual either). If it doesn't tell you how to use "+m", and even tells you _not_ to use it, maybe that is what it means to say? It doesn't mean "+m" doesn't actually do something. It also doesn't mean it does what you think it should do. It might do just that of course. But treating writing C code as an empirical science isn't such a smart idea. > And I didn't go whining about this ... you asked me. (I think I'd said > something to the effect of GCC docs are often wrong, No need to guess at what you said, even if you managed to delete your own mail already, there are plenty of free web-based archives around. You said: > See, "volatile" C keyword, for all it's ill-definition and dodgy > semantics, is still at least given somewhat of a treatment in the C > standard (whose quality is ... ummm, sadly not always good and clear, > but unsurprisingly, still about 5,482 orders-of-magnitude times > better than GCC docs). and that to me reads as complaining that the ISO C standard "isn't very good" and that the GCC documentation is 10**5482 times worse even. Which of course is hyperbole and cannot be true. It also isn't helpful in any way or form for anyone on this list. I call that whining. > which is true, Yes, documentation of that size often has shortcomings. No surprise there. However, great effort is made to make it better documentation, and especially to keep it up to date; if you find any errors or omissions, please report them. There are many ways how to do that, see the GCC homepage.</end-of-marketing-blurb> > but probably you feel saying that is "not allowed" on non-gcc lists?) You're allowed to say whatever you want. Let's have a quote again shall we? I said: > If you find any problems/shortcomings in the GCC documentation, > please file a PR, don't go whine on some unrelated mailing lists. > Thank you. I read that as a friendly request, not a prohibition. Well maybe not actually friendly, more a bit angry. A request, either way. > As for the "PR" "Problem report", a bugzilla ticket. Sorry for using terminology unknown to you. > you're requesting me to file with GCC for this, that > gcc-patches@ thread did precisely that Actually not -- PRs make sure issues aren't forgotten (although they might gather dust, sure). But yes, submitting patches is a Great Thing(tm). > and more (submitted a patch to > said documentation -- and no, saying "documentation might get added > later" is totally bogus and nonsensical -- documentation exists to > document current behaviour, not past). When code like you want to write becomes a supported feature, that will be reflected in the user manual. It is completely nonsensical to expect everything that is *not* a supported feature to be mentioned there. > I wouldn't have replied, really, if you weren't so provoking. Hey, maybe that character trait is good for something, then. Now to build a business plan around it... Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-18 5:18 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-18 5:18 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >> The documentation simply doesn't say "+m" is allowed. The code to >> allow it was added for the benefit of people who do not read the >> documentation. Documentation for "+m" might get added later if it >> is decided this [the code, not the documentation] is a sane thing >> to have (which isn't directly obvious). > > Huh? > > "If the (current) documentation doesn't match up with the (current) > code, then _at least one_ of them has to be (as of current) wrong." > > I wonder how could you even try to disagree with that. Easy. The GCC documentation you're referring to is the user's manual. See the blurb on the first page: "This manual documents how to use the GNU compilers, as well as their features and incompatibilities, and how to report bugs. It corresponds to GCC version 4.3.0. The internals of the GNU compilers, including how to port them to new targets and some information about how to write front ends for new languages, are documented in a separate manual." _How to use_. This documentation doesn't describe in minute detail everything the compiler does (see the source code for that -- no, it isn't described in the internals manual either). If it doesn't tell you how to use "+m", and even tells you _not_ to use it, maybe that is what it means to say? It doesn't mean "+m" doesn't actually do something. It also doesn't mean it does what you think it should do. It might do just that of course. But treating writing C code as an empirical science isn't such a smart idea. > And I didn't go whining about this ... you asked me. (I think I'd said > something to the effect of GCC docs are often wrong, No need to guess at what you said, even if you managed to delete your own mail already, there are plenty of free web-based archives around. You said: > See, "volatile" C keyword, for all it's ill-definition and dodgy > semantics, is still at least given somewhat of a treatment in the C > standard (whose quality is ... ummm, sadly not always good and clear, > but unsurprisingly, still about 5,482 orders-of-magnitude times > better than GCC docs). and that to me reads as complaining that the ISO C standard "isn't very good" and that the GCC documentation is 10**5482 times worse even. Which of course is hyperbole and cannot be true. It also isn't helpful in any way or form for anyone on this list. I call that whining. > which is true, Yes, documentation of that size often has shortcomings. No surprise there. However, great effort is made to make it better documentation, and especially to keep it up to date; if you find any errors or omissions, please report them. There are many ways how to do that, see the GCC homepage.</end-of-marketing-blurb> > but probably you feel saying that is "not allowed" on non-gcc lists?) You're allowed to say whatever you want. Let's have a quote again shall we? I said: > If you find any problems/shortcomings in the GCC documentation, > please file a PR, don't go whine on some unrelated mailing lists. > Thank you. I read that as a friendly request, not a prohibition. Well maybe not actually friendly, more a bit angry. A request, either way. > As for the "PR" "Problem report", a bugzilla ticket. Sorry for using terminology unknown to you. > you're requesting me to file with GCC for this, that > gcc-patches@ thread did precisely that Actually not -- PRs make sure issues aren't forgotten (although they might gather dust, sure). But yes, submitting patches is a Great Thing(tm). > and more (submitted a patch to > said documentation -- and no, saying "documentation might get added > later" is totally bogus and nonsensical -- documentation exists to > document current behaviour, not past). When code like you want to write becomes a supported feature, that will be reflected in the user manual. It is completely nonsensical to expect everything that is *not* a supported feature to be mentioned there. > I wouldn't have replied, really, if you weren't so provoking. Hey, maybe that character trait is good for something, then. Now to build a business plan around it... Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-18 5:18 ` Segher Boessenkool (?) @ 2007-08-18 13:20 ` Satyam Sharma -1 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-18 13:20 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, zlynx, Andrew Morton, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang [ LOL, you _are_ shockingly petty! ] On Sat, 18 Aug 2007, Segher Boessenkool wrote: > > > The documentation simply doesn't say "+m" is allowed. The code to > > > allow it was added for the benefit of people who do not read the > > > documentation. Documentation for "+m" might get added later if it > > > is decided this [the code, not the documentation] is a sane thing > > > to have (which isn't directly obvious). > > > > Huh? > > > > "If the (current) documentation doesn't match up with the (current) > > code, then _at least one_ of them has to be (as of current) wrong." > > > > I wonder how could you even try to disagree with that. > > Easy. > > The GCC documentation you're referring to is the user's manual. > See the blurb on the first page: > > "This manual documents how to use the GNU compilers, as well as their > features and incompatibilities, and how to report bugs. It corresponds > to GCC version 4.3.0. The internals of the GNU compilers, including > how to port them to new targets and some information about how to write > front ends for new languages, are documented in a separate manual." > > _How to use_. This documentation doesn't describe in minute detail > everything the compiler does (see the source code for that -- no, it > isn't described in the internals manual either). Wow, now that's a nice "disclaimer". By your (poor) standards of writing documentation, one can as well write any factually incorrect stuff that one wants in a document once you've got such a blurb in place :-) > If it doesn't tell you how to use "+m", and even tells you _not_ to > use it, maybe that is what it means to say? It doesn't mean "+m" > doesn't actually do something. It also doesn't mean it does what > you think it should do. It might do just that of course. But treating > writing C code as an empirical science isn't such a smart idea. Oh, really? Considering how much is (left out of being) documented, often one would reasonably have to experimentally see (with testcases) how the compiler behaves for some given code. Well, at least _I_ do it often (several others on this list do as well), and I think there's everything smart about it rather than having to read gcc sources -- I'd be surprised (unless you have infinite free time on your hands, which does look like teh case actually) if someone actually prefers reading gcc sources first to know what/how gcc does something for some given code, rather than simply write it out, compile and look the generated code (saves time for those who don't have an infinite amount of it). > > And I didn't go whining about this ... you asked me. (I think I'd said > > something to the effect of GCC docs are often wrong, > > No need to guess at what you said, even if you managed to delete > your own mail already, there are plenty of free web-based archives > around. You said: > > > See, "volatile" C keyword, for all it's ill-definition and dodgy > > semantics, is still at least given somewhat of a treatment in the C > > standard (whose quality is ... ummm, sadly not always good and clear, > > but unsurprisingly, still about 5,482 orders-of-magnitude times > > better than GCC docs). Try _reading_ what I said there, for a change, dude. I'd originally only said "unless GCC's docs is yet again wrong" ... then _you_ asked me what, after which this discussion began and I wrote the above [which I fully agree with -- so what if I used hyperbole in my sentence (yup, that was intended, and obviously, exaggeration), am I not even allowed to do that? Man, you're a Nazi or what ...] I didn't go whining about on my own as you'd had earlier suggested, until _you_ asked me. [ Ick, I somehow managed to reply this ... this is such a ... *disgustingly* petty argument you made here. ] > > which is true, > > Yes, documentation of that size often has shortcomings. No surprise > there. However, great effort is made to make it better documentation, > and especially to keep it up to date; if you find any errors or > omissions, please report them. There are many ways how to do that, > see the GCC homepage.</end-of-marketing-blurb> ^^^^^^^^^^^^^^^^^^^^^^ Looks like you even get paid :-) > > but probably you feel saying that is "not allowed" on non-gcc lists?) > > [amazingly pointless stuff snipped] > > > As for the "PR" > > you're requesting me to file with GCC for this, that > > gcc-patches@ thread did precisely that > > [more amazingly pointless stuff snipped] > > > and more (submitted a patch to > > said documentation -- and no, saying "documentation might get added > > later" is totally bogus and nonsensical -- documentation exists to > > document current behaviour, not past). > > When code like you want to write becomes a supported feature, that > will be reflected in the user manual. It is completely nonsensical > to expect everything that is *not* a supported feature to be mentioned > there. What crap. It is _perfectly reasonable_ to expect (current) documentation to keep up with (current) code behaviour. In fact trying to justify such a state is completely bogus and nonsensical. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 17:41 ` Segher Boessenkool 2007-08-17 18:38 ` Satyam Sharma @ 2007-09-10 18:59 ` Christoph Lameter 2007-09-10 20:54 ` Paul E. McKenney 2007-09-11 2:27 ` Segher Boessenkool 1 sibling, 2 replies; 333+ messages in thread From: Christoph Lameter @ 2007-09-10 18:59 UTC (permalink / raw) To: Segher Boessenkool Cc: Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Fri, 17 Aug 2007, Segher Boessenkool wrote: > "volatile" has nothing to do with reordering. atomic_dec() writes > to memory, so it _does_ have "volatile semantics", implicitly, as > long as the compiler cannot optimise the atomic variable away > completely -- any store counts as a side effect. Stores can be reordered. Only x86 has (mostly) implicit write ordering. So no atomic_dec has no volatile semantics and may be reordered on a variety of processors. Writes to memory may not follow code order on several processors. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 18:59 ` Christoph Lameter @ 2007-09-10 20:54 ` Paul E. McKenney 2007-09-10 21:36 ` Christoph Lameter 2007-09-11 2:27 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Paul E. McKenney @ 2007-09-10 20:54 UTC (permalink / raw) To: Christoph Lameter Cc: Segher Boessenkool, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, David Miller, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Mon, Sep 10, 2007 at 11:59:29AM -0700, Christoph Lameter wrote: > On Fri, 17 Aug 2007, Segher Boessenkool wrote: > > > "volatile" has nothing to do with reordering. atomic_dec() writes > > to memory, so it _does_ have "volatile semantics", implicitly, as > > long as the compiler cannot optimise the atomic variable away > > completely -- any store counts as a side effect. > > Stores can be reordered. Only x86 has (mostly) implicit write ordering. So > no atomic_dec has no volatile semantics and may be reordered on a variety > of processors. Writes to memory may not follow code order on several > processors. The one exception to this being the case where process-level code is communicating to an interrupt handler running on that same CPU -- on all CPUs that I am aware of, a given CPU always sees its own writes in order. Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 20:54 ` Paul E. McKenney @ 2007-09-10 21:36 ` Christoph Lameter 2007-09-10 21:50 ` Paul E. McKenney 0 siblings, 1 reply; 333+ messages in thread From: Christoph Lameter @ 2007-09-10 21:36 UTC (permalink / raw) To: Paul E. McKenney Cc: Segher Boessenkool, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, David Miller, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Mon, 10 Sep 2007, Paul E. McKenney wrote: > The one exception to this being the case where process-level code is > communicating to an interrupt handler running on that same CPU -- on > all CPUs that I am aware of, a given CPU always sees its own writes > in order. Yes but that is due to the code path effectively continuing in the interrupt handler. The cpu makes sure that op codes being executed always see memory in a consistent way. The basic ordering problem with out of order writes is therefore coming from other processors concurrently executing code and holding variables in registers that are modified elsewhere. The only solution that I know of are one or the other form of barrier. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 21:36 ` Christoph Lameter @ 2007-09-10 21:50 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-09-10 21:50 UTC (permalink / raw) To: Christoph Lameter Cc: Segher Boessenkool, Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, David Miller, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang On Mon, Sep 10, 2007 at 02:36:26PM -0700, Christoph Lameter wrote: > On Mon, 10 Sep 2007, Paul E. McKenney wrote: > > > The one exception to this being the case where process-level code is > > communicating to an interrupt handler running on that same CPU -- on > > all CPUs that I am aware of, a given CPU always sees its own writes > > in order. > > Yes but that is due to the code path effectively continuing in the > interrupt handler. The cpu makes sure that op codes being executed always > see memory in a consistent way. The basic ordering problem with out of > order writes is therefore coming from other processors concurrently > executing code and holding variables in registers that are modified > elsewhere. The only solution that I know of are one or the other form of > barrier. So we are agreed then -- volatile accesses may be of some assistance when interacting with interrupt handlers running on the same CPU (presumably when using per-CPU variables), but are generally useless when sharing variables among CPUs. Correct? Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-09-10 18:59 ` Christoph Lameter @ 2007-09-11 2:27 ` Segher Boessenkool 2007-09-11 2:27 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-09-11 2:27 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >> "volatile" has nothing to do with reordering. atomic_dec() writes >> to memory, so it _does_ have "volatile semantics", implicitly, as >> long as the compiler cannot optimise the atomic variable away >> completely -- any store counts as a side effect. > > Stores can be reordered. Only x86 has (mostly) implicit write ordering. > So no atomic_dec has no volatile semantics Read again: I said the C "volatile" construct has nothing to do with CPU memory access reordering. > and may be reordered on a variety > of processors. Writes to memory may not follow code order on several > processors. The _compiler_ isn't allowed to reorder things here. Yes, of course you do need stronger barriers for many purposes, volatile isn't all that useful you know. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-09-11 2:27 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-09-11 2:27 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, David Miller, Paul E. McKenney, Ilpo Järvinen, ak, cfriesen, rpjday, Netdev, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, Linus Torvalds, wensong, wjiang >> "volatile" has nothing to do with reordering. atomic_dec() writes >> to memory, so it _does_ have "volatile semantics", implicitly, as >> long as the compiler cannot optimise the atomic variable away >> completely -- any store counts as a side effect. > > Stores can be reordered. Only x86 has (mostly) implicit write ordering. > So no atomic_dec has no volatile semantics Read again: I said the C "volatile" construct has nothing to do with CPU memory access reordering. > and may be reordered on a variety > of processors. Writes to memory may not follow code order on several > processors. The _compiler_ isn't allowed to reorder things here. Yes, of course you do need stronger barriers for many purposes, volatile isn't all that useful you know. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 19:55 ` Chris Snook @ 2007-08-16 21:08 ` Luck, Tony 2007-08-16 21:08 ` Luck, Tony 1 sibling, 0 replies; 333+ messages in thread From: Luck, Tony @ 2007-08-16 21:08 UTC (permalink / raw) To: Chris Snook, Ilpo Järvinen Cc: Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher >> 6674: while (atomic_read(&j->DSPWrite) > 0) >> 6675- atomic_dec(&j->DSPWrite); > > If the maintainer of this code doesn't see a compelling reason to add > cpu_relax() in this loop, then it should be patched. Shouldn't it be just re-written without the loop: if ((tmp = atomic_read(&j->DSPWrite)) > 0) atomic_sub(&j->DSPWrite, tmp); Has all the same bugs, but runs much faster :-) -Tony ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH 0/24] make atomic_read() behave consistently across all architectures @ 2007-08-16 21:08 ` Luck, Tony 0 siblings, 0 replies; 333+ messages in thread From: Luck, Tony @ 2007-08-16 21:08 UTC (permalink / raw) To: Chris Snook, Ilpo Järvinen Cc: Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher >> 6674: while (atomic_read(&j->DSPWrite) > 0) >> 6675- atomic_dec(&j->DSPWrite); > > If the maintainer of this code doesn't see a compelling reason to add > cpu_relax() in this loop, then it should be patched. Shouldn't it be just re-written without the loop: if ((tmp = atomic_read(&j->DSPWrite)) > 0) atomic_sub(&j->DSPWrite, tmp); Has all the same bugs, but runs much faster :-) -Tony ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 14:48 ` Ilpo Järvinen 2007-08-16 16:19 ` Stefan Richter 2007-08-16 19:55 ` Chris Snook @ 2007-08-16 19:55 ` Chris Snook 2 siblings, 0 replies; 333+ messages in thread From: Chris Snook @ 2007-08-16 19:55 UTC (permalink / raw) To: Ilpo Järvinen Cc: Herbert Xu, Paul Mackerras, Satyam Sharma, Christoph Lameter, Paul E. McKenney, Stefan Richter, Linux Kernel Mailing List, linux-arch, Linus Torvalds, Netdev, Andrew Morton, ak, heiko.carstens, David Miller, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Ilpo Järvinen wrote: > On Thu, 16 Aug 2007, Herbert Xu wrote: > >> We've been through that already. If it's a busy-wait it >> should use cpu_relax. > > I looked around a bit by using some command lines and ended up wondering > if these are equal to busy-wait case (and should be fixed) or not: > > ./drivers/telephony/ixj.c > 6674: while (atomic_read(&j->DSPWrite) > 0) > 6675- atomic_dec(&j->DSPWrite); > > ...besides that, there are couple of more similar cases in the same file > (with braces)... atomic_dec() already has volatile behavior everywhere, so this is semantically okay, but this code (and any like it) should be calling cpu_relax() each iteration through the loop, unless there's a compelling reason not to. I'll allow that for some hardware drivers (possibly this one) such a compelling reason may exist, but hardware-independent core subsystems probably have no excuse. If the maintainer of this code doesn't see a compelling reason not to add cpu_relax() in this loop, then it should be patched. -- Chris ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 4:11 ` Paul Mackerras 2007-08-16 5:39 ` Herbert Xu @ 2007-08-16 18:54 ` Christoph Lameter 2007-08-16 20:07 ` Paul E. McKenney 1 sibling, 1 reply; 333+ messages in thread From: Christoph Lameter @ 2007-08-16 18:54 UTC (permalink / raw) To: Paul Mackerras Cc: Satyam Sharma, Herbert Xu, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, 16 Aug 2007, Paul Mackerras wrote: > The uses of atomic_read where one might want it to allow caching of > the result seem to me to fall into 3 categories: > > 1. Places that are buggy because of a race arising from the way it's > used. > > 2. Places where there is a race but it doesn't matter because we're > doing some clever trick. > > 3. Places where there is some locking in place that eliminates any > potential race. > > In case 1, adding volatile won't solve the race, of course, but it's > hard to argue that we shouldn't do something because it will slow down > buggy code. Case 2 is hopefully pretty rare and accompanied by large > comment blocks, and in those cases caching the result of atomic_read > explicitly in a local variable would probably make the code clearer. > And in case 3 there is no reason to use atomic_t at all; we might as > well just use an int. In 2 + 3 you may increment the atomic variable in some places. The value of the atomic variable may not matter because you only do optimizations. Checking a atomic_t for a definite state has to involve either some side conditions (lock only taken if refcount is <= 0 or so) or done by changing the state (see f.e. atomic_inc_unless_zero). > So I don't see any good reason to make the atomic API more complex by > having "volatile" and "non-volatile" versions of atomic_read. It > should just have the "volatile" behaviour. If you want to make it less complex then drop volatile which causes weird side effects without solving any problems as you just pointed out. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 18:54 ` Christoph Lameter @ 2007-08-16 20:07 ` Paul E. McKenney 0 siblings, 0 replies; 333+ messages in thread From: Paul E. McKenney @ 2007-08-16 20:07 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, Satyam Sharma, Herbert Xu, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher On Thu, Aug 16, 2007 at 11:54:54AM -0700, Christoph Lameter wrote: > On Thu, 16 Aug 2007, Paul Mackerras wrote: > > So I don't see any good reason to make the atomic API more complex by > > having "volatile" and "non-volatile" versions of atomic_read. It > > should just have the "volatile" behaviour. > > If you want to make it less complex then drop volatile which causes weird > side effects without solving any problems as you just pointed out. The other set of problems are communication between process context and interrupt/NMI handlers. Volatile does help here. And the performance impact of volatile is pretty near zero, so why have the non-volatile variant? Thanx, Paul ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 2:33 ` Satyam Sharma 2007-08-16 3:01 ` Satyam Sharma @ 2007-08-16 3:05 ` Paul Mackerras 2007-08-16 19:39 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Paul Mackerras @ 2007-08-16 3:05 UTC (permalink / raw) To: Satyam Sharma Cc: Herbert Xu, Christoph Lameter, Paul E. McKenney, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher Satyam Sharma writes: > I can't speak for this particular case, but there could be similar code > examples elsewhere, where we do the atomic ops on an atomic_t object > inside a higher-level locking scheme that would take care of the kind of > problem you're referring to here. It would be useful for such or similar > code if the compiler kept the value of that atomic object in a register. If there is a higher-level locking scheme then there is no point to using atomic_t variables. Atomic_t is specifically for the situation where multiple CPUs are updating a variable without locking. Paul. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:05 ` Paul Mackerras @ 2007-08-16 19:39 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 19:39 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> I can't speak for this particular case, but there could be similar >> code >> examples elsewhere, where we do the atomic ops on an atomic_t object >> inside a higher-level locking scheme that would take care of the kind >> of >> problem you're referring to here. It would be useful for such or >> similar >> code if the compiler kept the value of that atomic object in a >> register. > > If there is a higher-level locking scheme then there is no point to > using atomic_t variables. Atomic_t is specifically for the situation > where multiple CPUs are updating a variable without locking. And don't forget about the case where it is an I/O device that is updating the memory (in buffer descriptors or similar). The driver needs to do a "volatile" atomic read to get at the most recent version of that data, which can be important for optimising latency (or throughput even). There is no other way the kernel can get that info -- doing an MMIO read is way way too expensive. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 1:51 ` Paul Mackerras 2007-08-16 2:00 ` Herbert Xu @ 2007-08-16 2:07 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 2:07 UTC (permalink / raw) To: Paul Mackerras Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Satyam Sharma, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >> A volatile default would disable optimizations for atomic_read. >> atomic_read without volatile would allow for full optimization by the >> compiler. Seems that this is what one wants in many cases. > > Name one such case. > > An atomic_read should do a load from memory. If the programmer puts > an atomic_read() in the code then the compiler should emit a load for > it, not re-use a value returned by a previous atomic_read. I do not > believe it would ever be useful for the compiler to collapse two > atomic_read statements into a single load. An atomic_read() implemented as a "normal" C variable read would allow that read to be combined with another "normal" read from that variable. This could perhaps be marginally useful, although I'd bet you cannot see it unless counting cycles on a simulator or counting bits in the binary size. With an asm() implementation, the compiler can not do this; with a "volatile" implementation (either volatile variable or volatile-cast), this invokes undefined behaviour (in both C and GCC). Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 23:22 ` Paul Mackerras 2007-08-16 0:26 ` Christoph Lameter @ 2007-08-24 12:50 ` Denys Vlasenko 2007-08-24 17:15 ` Christoph Lameter 1 sibling, 1 reply; 333+ messages in thread From: Denys Vlasenko @ 2007-08-24 12:50 UTC (permalink / raw) To: Paul Mackerras Cc: Satyam Sharma, Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Thursday 16 August 2007 00:22, Paul Mackerras wrote: > Satyam Sharma writes: > In the kernel we use atomic variables in precisely those situations > where a variable is potentially accessed concurrently by multiple > CPUs, and where each CPU needs to see updates done by other CPUs in a > timely fashion. That is what they are for. Therefore the compiler > must not cache values of atomic variables in registers; each > atomic_read must result in a load and each atomic_set must result in a > store. Anything else will just lead to subtle bugs. Amen. -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-24 12:50 ` Denys Vlasenko @ 2007-08-24 17:15 ` Christoph Lameter 2007-08-24 20:21 ` Denys Vlasenko 0 siblings, 1 reply; 333+ messages in thread From: Christoph Lameter @ 2007-08-24 17:15 UTC (permalink / raw) To: Denys Vlasenko Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Fri, 24 Aug 2007, Denys Vlasenko wrote: > On Thursday 16 August 2007 00:22, Paul Mackerras wrote: > > Satyam Sharma writes: > > In the kernel we use atomic variables in precisely those situations > > where a variable is potentially accessed concurrently by multiple > > CPUs, and where each CPU needs to see updates done by other CPUs in a > > timely fashion. That is what they are for. Therefore the compiler > > must not cache values of atomic variables in registers; each > > atomic_read must result in a load and each atomic_set must result in a > > store. Anything else will just lead to subtle bugs. > > Amen. A "timely" fashion? One cannot rely on something like that when coding. The visibility of updates is insured by barriers and not by some fuzzy notion of "timeliness". ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-24 17:15 ` Christoph Lameter @ 2007-08-24 20:21 ` Denys Vlasenko 0 siblings, 0 replies; 333+ messages in thread From: Denys Vlasenko @ 2007-08-24 20:21 UTC (permalink / raw) To: Christoph Lameter Cc: Paul Mackerras, Satyam Sharma, Stefan Richter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Friday 24 August 2007 18:15, Christoph Lameter wrote: > On Fri, 24 Aug 2007, Denys Vlasenko wrote: > > On Thursday 16 August 2007 00:22, Paul Mackerras wrote: > > > Satyam Sharma writes: > > > In the kernel we use atomic variables in precisely those situations > > > where a variable is potentially accessed concurrently by multiple > > > CPUs, and where each CPU needs to see updates done by other CPUs in a > > > timely fashion. That is what they are for. Therefore the compiler > > > must not cache values of atomic variables in registers; each > > > atomic_read must result in a load and each atomic_set must result in a > > > store. Anything else will just lead to subtle bugs. > > > > Amen. > > A "timely" fashion? One cannot rely on something like that when coding. > The visibility of updates is insured by barriers and not by some fuzzy > notion of "timeliness". But here you do have some notion of time: while (atomic_read(&x)) continue; "continue when other CPU(s) decrement it down to zero". If "read" includes an insn which accesses RAM, you will see "new" value sometime after other CPU decrements it. "Sometime after" is on the order of nanoseconds here. It is a valid concept of time, right? The whole confusion is about whether atomic_read implies "read from RAM" or not. I am in a camp which thinks it does. You are in an opposite one. We just need a less ambiguous name. What about this: /** * atomic_read - read atomic variable * @v: pointer of type atomic_t * * Atomically reads the value of @v. * No compiler barrier implied. */ #define atomic_read(v) ((v)->counter) +/** + * atomic_read_uncached - read atomic variable from memory + * @v: pointer of type atomic_t + * + * Atomically reads the value of @v. This is guaranteed to emit an insn + * which accesses memory, atomically. No ordering guarantees! + */ +#define atomic_read_uncached(v) asm_or_volatile_ptr_magic(v) I was thinking of s/atomic_read/atomic_get/ too, but it implies "taking" atomic a-la get_cpu()... -- vda ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 12:31 ` Satyam Sharma ` (2 preceding siblings ...) 2007-08-15 23:22 ` Paul Mackerras @ 2007-08-16 3:37 ` Bill Fink 2007-08-16 5:20 ` Satyam Sharma 3 siblings, 1 reply; 333+ messages in thread From: Bill Fink @ 2007-08-16 3:37 UTC (permalink / raw) To: Satyam Sharma Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Wed, 15 Aug 2007, Satyam Sharma wrote: > (C) > $ cat tp3.c > int a; > > void func(void) > { > *(volatile int *)&a = 10; > *(volatile int *)&a = 20; > } > $ gcc -Os -S tp3.c > $ cat tp3.s > ... > movl $10, a > movl $20, a > ... I'm curious about one minor tangential point. Why, instead of: b = *(volatile int *)&a; why can't this just be expressed as: b = (volatile int)a; Isn't it the contents of a that's volatile, i.e. it's value can change invisibly to the compiler, and that's why you want to force a read from memory? Why do you need the "*(volatile int *)&" construct? -Bill ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 3:37 ` Bill Fink @ 2007-08-16 5:20 ` Satyam Sharma 2007-08-16 5:57 ` Satyam Sharma 2007-08-16 20:50 ` Segher Boessenkool 0 siblings, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 5:20 UTC (permalink / raw) To: Bill Fink Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney Hi Bill, On Wed, 15 Aug 2007, Bill Fink wrote: > On Wed, 15 Aug 2007, Satyam Sharma wrote: > > > (C) > > $ cat tp3.c > > int a; > > > > void func(void) > > { > > *(volatile int *)&a = 10; > > *(volatile int *)&a = 20; > > } > > $ gcc -Os -S tp3.c > > $ cat tp3.s > > ... > > movl $10, a > > movl $20, a > > ... > > I'm curious about one minor tangential point. Why, instead of: > > b = *(volatile int *)&a; > > why can't this just be expressed as: > > b = (volatile int)a; > > Isn't it the contents of a that's volatile, i.e. it's value can change > invisibly to the compiler, and that's why you want to force a read from > memory? Why do you need the "*(volatile int *)&" construct? "b = (volatile int)a;" doesn't help us because a cast to a qualified type has the same effect as a cast to an unqualified version of that type, as mentioned in 6.5.4:4 (footnote 86) of the standard. Note that "volatile" is a type-qualifier, not a type itself, so a cast of the _object_ itself to a qualified-type i.e. (volatile int) would not make the access itself volatile-qualified. To serve our purposes, it is necessary for us to take the address of this (non-volatile) object, cast the resulting _pointer_ to the corresponding volatile-qualified pointer-type, and then dereference it. This makes that particular _access_ be volatile-qualified, without the object itself being such. Also note that the (dereferenced) result is also a valid lvalue and hence can be used in "*(volatile int *)&a = b;" kind of construction (which we use for the atomic_set case). Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 5:20 ` Satyam Sharma @ 2007-08-16 5:57 ` Satyam Sharma 2007-08-16 9:25 ` Satyam Sharma 2007-08-16 21:00 ` Segher Boessenkool 2007-08-16 20:50 ` Segher Boessenkool 1 sibling, 2 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 5:57 UTC (permalink / raw) To: Bill Fink Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Thu, 16 Aug 2007, Satyam Sharma wrote: > Hi Bill, > > > On Wed, 15 Aug 2007, Bill Fink wrote: > > > On Wed, 15 Aug 2007, Satyam Sharma wrote: > > > > > (C) > > > $ cat tp3.c > > > int a; > > > > > > void func(void) > > > { > > > *(volatile int *)&a = 10; > > > *(volatile int *)&a = 20; > > > } > > > $ gcc -Os -S tp3.c > > > $ cat tp3.s > > > ... > > > movl $10, a > > > movl $20, a > > > ... > > > > I'm curious about one minor tangential point. Why, instead of: > > > > b = *(volatile int *)&a; > > > > why can't this just be expressed as: > > > > b = (volatile int)a; > > > > Isn't it the contents of a that's volatile, i.e. it's value can change > > invisibly to the compiler, and that's why you want to force a read from > > memory? Why do you need the "*(volatile int *)&" construct? > > "b = (volatile int)a;" doesn't help us because a cast to a qualified type > has the same effect as a cast to an unqualified version of that type, as > mentioned in 6.5.4:4 (footnote 86) of the standard. Note that "volatile" > is a type-qualifier, not a type itself, so a cast of the _object_ itself > to a qualified-type i.e. (volatile int) would not make the access itself > volatile-qualified. > > To serve our purposes, it is necessary for us to take the address of this > (non-volatile) object, cast the resulting _pointer_ to the corresponding > volatile-qualified pointer-type, and then dereference it. This makes that > particular _access_ be volatile-qualified, without the object itself being > such. Also note that the (dereferenced) result is also a valid lvalue and > hence can be used in "*(volatile int *)&a = b;" kind of construction > (which we use for the atomic_set case). Here, I should obviously admit that the semantics of *(volatile int *)& aren't any neater or well-defined in the _language standard_ at all. The standard does say (verbatim) "precisely what constitutes as access to object of volatile-qualified type is implementation-defined", but GCC does help us out here by doing the right thing. Accessing the non-volatile object there using the volatile-qualified pointer-type cast makes GCC treat the object stored at that memory address itself as if it were a volatile object, thus making the access end up having what we're calling as "volatility" semantics here. Honestly, given such confusion, and the propensity of the "volatile" type-qualifier keyword to be ill-defined (or at least poorly understood, often inconsistently implemented), I'd (again) express my opinion that it would be best to avoid its usage, given other alternatives do exist. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 5:57 ` Satyam Sharma @ 2007-08-16 9:25 ` Satyam Sharma 2007-08-16 21:00 ` Segher Boessenkool 1 sibling, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-16 9:25 UTC (permalink / raw) To: Bill Fink Cc: Stefan Richter, Christoph Lameter, Chris Snook, Linux Kernel Mailing List, linux-arch, Linus Torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney [ Bill tells me in private communication he gets this already, but I think it's more complicated than the shoddy explanation I'd made earlier so would wish to make this clearer in detail one last time, for the benefit of others listening in or reading the archives. ] On Thu, 16 Aug 2007, Satyam Sharma wrote: > On Thu, 16 Aug 2007, Satyam Sharma wrote: > [...] > > On Wed, 15 Aug 2007, Bill Fink wrote: > > > [...] > > > I'm curious about one minor tangential point. Why, instead of: > > > > > > b = *(volatile int *)&a; > > > > > > why can't this just be expressed as: > > > > > > b = (volatile int)a; > > > > > > Isn't it the contents of a that's volatile, i.e. it's value can change > > > invisibly to the compiler, and that's why you want to force a read from > > > memory? Why do you need the "*(volatile int *)&" construct? > > > > "b = (volatile int)a;" doesn't help us because a cast to a qualified type > > has the same effect as a cast to an unqualified version of that type, as > > mentioned in 6.5.4:4 (footnote 86) of the standard. Note that "volatile" > > is a type-qualifier, not a type itself, so a cast of the _object_ itself > > to a qualified-type i.e. (volatile int) would not make the access itself > > volatile-qualified. Casts don't produce lvalues, and the cast ((volatile int)a) does not produce the object-int-a-qualified-as-"volatile" -- in fact, the result of the above cast is whatever is the _value_ of "int a", with the access to that object having _already_ taken place, as per the actual type-qualification of the object (that was originally declared as being _non-volatile_, in fact). Hence, defining atomic_read() as: #define atomic_read(v) ((volatile int)((v)->counter)) would be buggy and not give "volatility" semantics at all, unless the "counter" object itself isn't volatile-qualified already (which it isn't). The result of the cast itself being the _value_ of the int object, and not the object itself (i.e., not an lvalue), is thereby independent of type-qualification in that cast itself (it just wouldn't make any difference), hence the "cast to a qualified type has the same effect as a cast to an unqualified version of that type" bit in section 6.5.4:4 of the standard. > > To serve our purposes, it is necessary for us to take the address of this > > (non-volatile) object, cast the resulting _pointer_ to the corresponding > > volatile-qualified pointer-type, and then dereference it. This makes that > > particular _access_ be volatile-qualified, without the object itself being > > such. Also note that the (dereferenced) result is also a valid lvalue and > > hence can be used in "*(volatile int *)&a = b;" kind of construction > > (which we use for the atomic_set case). Dereferencing using the *(pointer-type-cast)& construct, OTOH, serves us well: #define atomic_read(v) (*(volatile int *)&(v)->counter) Firstly, note that the cast here being (volatile int *) and not (int * volatile) qualifies the type of the _object_ being pointed to by the pointer in question as being volatile-qualified, and not the pointer itself (6.2.5:27 of the standard, and 6.3.2.3:2 allows us to convert from a pointer-to-non-volatile-qualified-int to a pointer-to- volatile-qualified-int, which suits us just fine) -- but note that the _access_ to that address itself has not yet occurred. _After_ specifying the memory address as containing a volatile-qualified- int-type object, (and GCC co-operates as mentioned below), we proceed to dereference it, which is when the _actual access_ occurs, therefore with "volatility" semantics this time. Interesting. > Here, I should obviously admit that the semantics of *(volatile int *)& > aren't any neater or well-defined in the _language standard_ at all. The > standard does say (verbatim) "precisely what constitutes as access to > object of volatile-qualified type is implementation-defined", but GCC > does help us out here by doing the right thing. Accessing the non-volatile > object there using the volatile-qualified pointer-type cast makes GCC > treat the object stored at that memory address itself as if it were a > volatile object, thus making the access end up having what we're calling > as "volatility" semantics here. > > Honestly, given such confusion, and the propensity of the "volatile" > type-qualifier keyword to be ill-defined (or at least poorly understood, > often inconsistently implemented), I'd (again) express my opinion that it > would be best to avoid its usage, given other alternatives do exist. Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 5:57 ` Satyam Sharma 2007-08-16 9:25 ` Satyam Sharma @ 2007-08-16 21:00 ` Segher Boessenkool 2007-08-17 4:32 ` Satyam Sharma 1 sibling, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 21:00 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Bill Fink, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang > Here, I should obviously admit that the semantics of *(volatile int *)& > aren't any neater or well-defined in the _language standard_ at all. > The > standard does say (verbatim) "precisely what constitutes as access to > object of volatile-qualified type is implementation-defined", but GCC > does help us out here by doing the right thing. Where do you get that idea? GCC manual, section 6.1, "When is a Volatile Object Accessed?" doesn't say anything of the kind. PR33053 and some others. > Honestly, given such confusion, and the propensity of the "volatile" > type-qualifier keyword to be ill-defined (or at least poorly > understood, > often inconsistently implemented), I'd (again) express my opinion that > it > would be best to avoid its usage, given other alternatives do exist. Yeah. Or we can have an email thread like this every time someone proposes a patch that uses an atomic variable ;-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 21:00 ` Segher Boessenkool @ 2007-08-17 4:32 ` Satyam Sharma 2007-08-17 22:38 ` Segher Boessenkool 0 siblings, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 4:32 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Bill Fink, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Thu, 16 Aug 2007, Segher Boessenkool wrote: > > Here, I should obviously admit that the semantics of *(volatile int *)& > > aren't any neater or well-defined in the _language standard_ at all. The > > standard does say (verbatim) "precisely what constitutes as access to > > object of volatile-qualified type is implementation-defined", but GCC > > does help us out here by doing the right thing. > > Where do you get that idea? Try a testcase (experimentally verify). > GCC manual, section 6.1, "When > is a Volatile Object Accessed?" doesn't say anything of the > kind. True, "implementation-defined" as per the C standard _is_ supposed to mean "unspecified behaviour where each implementation documents how the choice is made". So ok, probably GCC isn't "documenting" this implementation-defined behaviour which it is supposed to, but can't really fault them much for this, probably. ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 4:32 ` Satyam Sharma @ 2007-08-17 22:38 ` Segher Boessenkool 2007-08-18 14:42 ` Satyam Sharma 0 siblings, 1 reply; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 22:38 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Bill Fink, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang >>> Here, I should obviously admit that the semantics of *(volatile int >>> *)& >>> aren't any neater or well-defined in the _language standard_ at all. >>> The >>> standard does say (verbatim) "precisely what constitutes as access to >>> object of volatile-qualified type is implementation-defined", but GCC >>> does help us out here by doing the right thing. >> >> Where do you get that idea? > > Try a testcase (experimentally verify). That doesn't prove anything. Experiments can only disprove things. >> GCC manual, section 6.1, "When >> is a Volatile Object Accessed?" doesn't say anything of the >> kind. > > True, "implementation-defined" as per the C standard _is_ supposed to > mean > "unspecified behaviour where each implementation documents how the > choice > is made". So ok, probably GCC isn't "documenting" this > implementation-defined behaviour which it is supposed to, but can't > really > fault them much for this, probably. GCC _is_ documenting this, namely in this section 6.1. It doesn't mention volatile-casted stuff. Draw your own conclusions. Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 22:38 ` Segher Boessenkool @ 2007-08-18 14:42 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-18 14:42 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Bill Fink, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang On Sat, 18 Aug 2007, Segher Boessenkool wrote: > > > GCC manual, section 6.1, "When ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > is a Volatile Object Accessed?" doesn't say anything of the ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > > kind. ^^^^^ > > True, "implementation-defined" as per the C standard _is_ supposed to mean ^^^^^ > > "unspecified behaviour where each implementation documents how the choice > > is made". So ok, probably GCC isn't "documenting" this ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > implementation-defined behaviour which it is supposed to, but can't really > > fault them much for this, probably. > > GCC _is_ documenting this, namely in this section 6.1. (Again totally petty, but) Yes, but ... > It doesn't ^^^^^^^^^^ > mention volatile-casted stuff. Draw your own conclusions. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... exactly. So that's why I said "GCC isn't documenting _this_". Man, try _reading_ mails before replying to them ... ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 5:20 ` Satyam Sharma 2007-08-16 5:57 ` Satyam Sharma @ 2007-08-16 20:50 ` Segher Boessenkool 2007-08-16 22:40 ` David Schwartz 2007-08-17 4:24 ` Satyam Sharma 1 sibling, 2 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-16 20:50 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Bill Fink, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang > Note that "volatile" > is a type-qualifier, not a type itself, so a cast of the _object_ > itself > to a qualified-type i.e. (volatile int) would not make the access > itself > volatile-qualified. There is no such thing as "volatile-qualified access" defined anywhere; there only is the concept of a "volatile-qualified *object*". > To serve our purposes, it is necessary for us to take the address of > this > (non-volatile) object, cast the resulting _pointer_ to the > corresponding > volatile-qualified pointer-type, and then dereference it. This makes > that > particular _access_ be volatile-qualified, without the object itself > being > such. Also note that the (dereferenced) result is also a valid lvalue > and > hence can be used in "*(volatile int *)&a = b;" kind of construction > (which we use for the atomic_set case). There is a quite convincing argument that such an access _is_ an access to a volatile object; see GCC PR21568 comment #9. This probably isn't the last word on the matter though... Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 20:50 ` Segher Boessenkool @ 2007-08-16 22:40 ` David Schwartz 2007-08-17 4:36 ` Satyam Sharma 2007-08-17 4:24 ` Satyam Sharma 1 sibling, 1 reply; 333+ messages in thread From: David Schwartz @ 2007-08-16 22:40 UTC (permalink / raw) To: Linux-Kernel@Vger. Kernel. Org > There is a quite convincing argument that such an access _is_ an > access to a volatile object; see GCC PR21568 comment #9. This > probably isn't the last word on the matter though... I find this argument completely convincing and retract the contrary argument that I've made many times in this forum and others. You learn something new every day. Just in case it wasn't clear: int i; *(volatile int *)&i=2; In this case, there *is* an access to a volatile object. This is the end result of the the standard's definition of what it means to apply the 'volatile int *' cast to '&i' and then apply the '*' operator to the result and use it as an lvalue. C does not define the type of an object by how it is defined but by how it is accessed! DS ^ permalink raw reply [flat|nested] 333+ messages in thread
* RE: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 22:40 ` David Schwartz @ 2007-08-17 4:36 ` Satyam Sharma 0 siblings, 0 replies; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 4:36 UTC (permalink / raw) To: David Schwartz; +Cc: Linux-Kernel@Vger. Kernel. Org [ Your mailer drops Cc: lists, munges headers, does all sorts of badness. Please fix that. ] On Thu, 16 Aug 2007, David Schwartz wrote: > > > There is a quite convincing argument that such an access _is_ an > > access to a volatile object; see GCC PR21568 comment #9. This > > probably isn't the last word on the matter though... > > I find this argument completely convincing and retract the contrary argument > that I've made many times in this forum and others. You learn something new > every day. > > Just in case it wasn't clear: > int i; > *(volatile int *)&i=2; > > In this case, there *is* an access to a volatile object. This is the end > result of the the standard's definition of what it means to apply the > 'volatile int *' cast to '&i' and then apply the '*' operator to the result > and use it as an lvalue. True, see my last mail in this sub-thread that explains precisely this :-) Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-16 20:50 ` Segher Boessenkool 2007-08-16 22:40 ` David Schwartz @ 2007-08-17 4:24 ` Satyam Sharma 2007-08-17 22:34 ` Segher Boessenkool 1 sibling, 1 reply; 333+ messages in thread From: Satyam Sharma @ 2007-08-17 4:24 UTC (permalink / raw) To: Segher Boessenkool Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Bill Fink, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang, davids On Thu, 16 Aug 2007, Segher Boessenkool wrote: > > Note that "volatile" > > is a type-qualifier, not a type itself, so a cast of the _object_ itself > > to a qualified-type i.e. (volatile int) would not make the access itself > > volatile-qualified. > > There is no such thing as "volatile-qualified access" defined > anywhere; there only is the concept of a "volatile-qualified > *object*". Sure, "volatile-qualified access" was not some standard term I used there. Just something to mean "an access that would make the compiler treat the object at that memory as if it were an object with a volatile-qualified type". Now the second wording *IS* technically correct, but come on, it's 24 words long whereas the original one was 3 -- and hopefully anybody reading the shorter phrase *would* have known anyway what was meant, without having to be pedantic about it :-) Satyam ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-17 4:24 ` Satyam Sharma @ 2007-08-17 22:34 ` Segher Boessenkool 0 siblings, 0 replies; 333+ messages in thread From: Segher Boessenkool @ 2007-08-17 22:34 UTC (permalink / raw) To: Satyam Sharma Cc: Christoph Lameter, heiko.carstens, horms, Stefan Richter, Bill Fink, Linux Kernel Mailing List, Paul E. McKenney, netdev, ak, cfriesen, rpjday, jesper.juhl, linux-arch, Andrew Morton, zlynx, davids, schwidefsky, Chris Snook, Herbert Xu, davem, Linus Torvalds, wensong, wjiang > Now the second wording *IS* technically correct, but come on, it's > 24 words long whereas the original one was 3 -- and hopefully anybody > reading the shorter phrase *would* have known anyway what was meant, > without having to be pedantic about it :-) Well you were talking pretty formal (and detailed) stuff, so IMHO it's good to have that exactly correct. Sure it's nicer to use small words most of the time :-) Segher ^ permalink raw reply [flat|nested] 333+ messages in thread
* Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures 2007-08-15 10:35 ` Stefan Richter 2007-08-15 12:04 ` Herbert Xu 2007-08-15 12:31 ` Satyam Sharma @ 2007-08-15 19:59 ` Christoph Lameter 2 siblings, 0 replies; 333+ messages in thread From: Christoph Lameter @ 2007-08-15 19:59 UTC (permalink / raw) To: Stefan Richter Cc: Satyam Sharma, Chris Snook, Linux Kernel Mailing List, linux-arch, torvalds, netdev, Andrew Morton, ak, heiko.carstens, davem, schwidefsky, wensong, horms, wjiang, cfriesen, zlynx, rpjday, jesper.juhl, segher, Herbert Xu, Paul E. McKenney On Wed, 15 Aug 2007, Stefan Richter wrote: > LDD3 says on page 125: "The following operations are defined for the > type [atomic_t] and are guaranteed to be atomic with respect to all > processors of an SMP computer." > > Doesn't "atomic WRT all processors" require volatility? Atomic operations only require exclusive access to the cacheline while the value is modified. ^ permalink raw reply [flat|nested] 333+ messages in thread
end of thread, other threads:[~2007-09-11 19:35 UTC | newest] Thread overview: 333+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2007-08-09 13:14 [PATCH 0/24] make atomic_read() behave consistently across all architectures Chris Snook 2007-08-09 12:41 ` Arnd Bergmann 2007-08-09 14:29 ` Chris Snook 2007-08-09 15:30 ` Arnd Bergmann 2007-08-14 22:31 ` Christoph Lameter 2007-08-14 22:45 ` Chris Snook 2007-08-14 22:51 ` Christoph Lameter 2007-08-14 23:08 ` Satyam Sharma 2007-08-14 23:04 ` Chris Snook 2007-08-14 23:14 ` Christoph Lameter 2007-08-15 6:49 ` Herbert Xu 2007-08-15 6:49 ` Herbert Xu 2007-08-15 8:18 ` Heiko Carstens 2007-08-15 13:53 ` Stefan Richter 2007-08-15 14:35 ` Satyam Sharma 2007-08-15 14:52 ` Herbert Xu 2007-08-15 16:09 ` Stefan Richter 2007-08-15 16:27 ` Paul E. McKenney 2007-08-15 17:13 ` Satyam Sharma 2007-08-15 18:31 ` Segher Boessenkool 2007-08-15 18:57 ` Paul E. McKenney 2007-08-15 19:54 ` Satyam Sharma 2007-08-15 20:17 ` Paul E. McKenney 2007-08-15 20:52 ` Segher Boessenkool 2007-08-15 22:42 ` Paul E. McKenney 2007-08-15 20:47 ` Segher Boessenkool 2007-08-16 0:36 ` Satyam Sharma 2007-08-16 0:36 ` (unknown) Satyam Sharma 2007-08-16 0:32 ` your mail Herbert Xu 2007-08-16 0:58 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Satyam Sharma 2007-08-16 0:51 ` Herbert Xu 2007-08-16 1:18 ` Satyam Sharma 2007-08-16 1:38 ` Segher Boessenkool 2007-08-15 21:05 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Segher Boessenkool 2007-08-15 22:44 ` Paul E. McKenney 2007-08-16 1:23 ` Segher Boessenkool 2007-08-16 2:22 ` Paul E. McKenney 2007-08-15 19:58 ` Stefan Richter 2007-08-16 0:39 ` [PATCH] i386: Fix a couple busy loops in mach_wakecpu.h:wait_for_init_deassert() Satyam Sharma 2007-08-24 11:59 ` Denys Vlasenko 2007-08-24 12:07 ` Andi Kleen 2007-08-24 12:12 ` Kenn Humborg 2007-08-24 12:12 ` Kenn Humborg 2007-08-24 14:25 ` Denys Vlasenko 2007-08-24 17:34 ` Linus Torvalds 2007-08-24 13:30 ` Satyam Sharma 2007-08-24 17:06 ` Christoph Lameter 2007-08-24 20:26 ` Denys Vlasenko 2007-08-24 20:34 ` Chris Snook 2007-08-24 16:19 ` Luck, Tony 2007-08-24 16:19 ` Luck, Tony 2007-08-15 16:13 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Chris Snook 2007-08-15 23:40 ` Herbert Xu 2007-08-15 23:51 ` Paul E. McKenney 2007-08-16 1:30 ` Segher Boessenkool 2007-08-16 2:30 ` Paul E. McKenney 2007-08-16 19:33 ` Segher Boessenkool 2007-08-16 1:26 ` Segher Boessenkool 2007-08-16 2:23 ` Nick Piggin 2007-08-16 19:32 ` Segher Boessenkool 2007-08-17 2:19 ` Nick Piggin 2007-08-17 3:16 ` Paul Mackerras 2007-08-17 3:32 ` Nick Piggin 2007-08-17 3:50 ` Linus Torvalds 2007-08-17 23:59 ` Paul E. McKenney 2007-08-18 0:09 ` Herbert Xu 2007-08-18 1:08 ` Paul E. McKenney 2007-08-18 1:24 ` Christoph Lameter 2007-08-18 1:41 ` Satyam Sharma 2007-08-18 4:13 ` Linus Torvalds 2007-08-18 13:36 ` Satyam Sharma 2007-08-18 21:54 ` Paul E. McKenney 2007-08-18 22:41 ` Linus Torvalds 2007-08-18 23:19 ` Paul E. McKenney 2007-08-24 12:19 ` Denys Vlasenko 2007-08-24 17:19 ` Linus Torvalds 2007-08-18 21:56 ` Paul E. McKenney 2007-08-20 13:31 ` Chris Snook 2007-08-20 22:04 ` Segher Boessenkool 2007-08-20 22:48 ` Russell King 2007-08-20 23:02 ` Segher Boessenkool 2007-08-21 0:05 ` Paul E. McKenney 2007-08-21 7:08 ` Russell King 2007-08-21 7:05 ` Russell King 2007-08-21 9:33 ` Paul Mackerras 2007-08-21 11:37 ` Andi Kleen 2007-08-21 14:48 ` Segher Boessenkool 2007-08-21 16:16 ` Paul E. McKenney 2007-08-21 22:51 ` Valdis.Kletnieks 2007-08-22 0:50 ` Paul E. McKenney 2007-08-22 21:38 ` Adrian Bunk 2007-08-21 14:39 ` Segher Boessenkool 2007-08-17 3:42 ` Linus Torvalds 2007-08-17 5:18 ` Paul E. McKenney 2007-08-17 5:56 ` Satyam Sharma 2007-08-17 7:26 ` Nick Piggin 2007-08-17 8:47 ` Satyam Sharma 2007-08-17 9:15 ` Nick Piggin 2007-08-17 10:12 ` Satyam Sharma 2007-08-17 12:14 ` Nick Piggin 2007-08-17 13:05 ` Satyam Sharma 2007-08-17 9:48 ` Paul Mackerras 2007-08-17 10:23 ` Satyam Sharma 2007-08-17 22:49 ` Segher Boessenkool 2007-08-17 23:51 ` Satyam Sharma 2007-08-17 23:55 ` Segher Boessenkool 2007-08-17 6:42 ` Geert Uytterhoeven 2007-08-17 8:52 ` Andi Kleen 2007-08-17 10:08 ` Satyam Sharma 2007-08-17 22:29 ` Segher Boessenkool 2007-08-17 17:37 ` Segher Boessenkool 2007-08-14 23:26 ` Paul E. McKenney 2007-08-15 10:35 ` Stefan Richter 2007-08-15 12:04 ` Herbert Xu 2007-08-15 12:31 ` Satyam Sharma 2007-08-15 13:08 ` Stefan Richter 2007-08-15 13:11 ` Stefan Richter 2007-08-15 13:47 ` Satyam Sharma 2007-08-15 14:25 ` Paul E. McKenney 2007-08-15 15:33 ` Herbert Xu 2007-08-15 16:08 ` Paul E. McKenney 2007-08-15 17:18 ` Satyam Sharma 2007-08-15 17:33 ` Paul E. McKenney 2007-08-15 18:05 ` Satyam Sharma 2007-08-15 17:55 ` Satyam Sharma 2007-08-15 19:07 ` Paul E. McKenney 2007-08-15 21:07 ` Segher Boessenkool 2007-08-15 20:58 ` Segher Boessenkool 2007-08-15 18:19 ` David Howells 2007-08-15 18:45 ` Paul E. McKenney 2007-08-15 23:41 ` Herbert Xu 2007-08-15 23:53 ` Paul E. McKenney 2007-08-16 0:12 ` Herbert Xu 2007-08-16 0:23 ` Paul E. McKenney 2007-08-16 0:30 ` Herbert Xu 2007-08-16 0:49 ` Paul E. McKenney 2007-08-16 0:53 ` Herbert Xu 2007-08-16 1:14 ` Paul E. McKenney 2007-08-15 18:31 ` Segher Boessenkool 2007-08-15 19:40 ` Satyam Sharma 2007-08-15 20:42 ` Segher Boessenkool 2007-08-16 1:23 ` Satyam Sharma 2007-08-15 23:22 ` Paul Mackerras 2007-08-16 0:26 ` Christoph Lameter 2007-08-16 0:34 ` Paul Mackerras 2007-08-16 0:40 ` Christoph Lameter 2007-08-16 0:39 ` Paul E. McKenney 2007-08-16 0:42 ` Christoph Lameter 2007-08-16 0:53 ` Paul E. McKenney 2007-08-16 0:59 ` Christoph Lameter 2007-08-16 1:14 ` Paul E. McKenney 2007-08-16 1:41 ` Christoph Lameter 2007-08-16 2:15 ` Satyam Sharma 2007-08-16 2:08 ` Herbert Xu 2007-08-16 2:18 ` Christoph Lameter 2007-08-16 3:23 ` Paul Mackerras 2007-08-16 3:33 ` Herbert Xu 2007-08-16 3:48 ` Paul Mackerras 2007-08-16 4:03 ` Herbert Xu 2007-08-16 4:34 ` Paul Mackerras 2007-08-16 5:37 ` Herbert Xu 2007-08-16 6:00 ` Paul Mackerras 2007-08-16 18:50 ` Christoph Lameter 2007-08-16 18:48 ` Christoph Lameter 2007-08-16 19:44 ` Segher Boessenkool 2007-08-16 2:18 ` Chris Friesen 2007-08-16 2:32 ` Paul E. McKenney 2007-08-16 1:51 ` Paul Mackerras 2007-08-16 2:00 ` Herbert Xu 2007-08-16 2:05 ` Paul Mackerras 2007-08-16 2:11 ` Herbert Xu 2007-08-16 2:35 ` Paul E. McKenney 2007-08-16 3:15 ` Paul Mackerras 2007-08-16 3:43 ` Herbert Xu 2007-08-16 2:15 ` Christoph Lameter 2007-08-16 2:17 ` Christoph Lameter 2007-08-16 2:33 ` Satyam Sharma 2007-08-16 3:01 ` Satyam Sharma 2007-08-16 4:11 ` Paul Mackerras 2007-08-16 5:39 ` Herbert Xu 2007-08-16 6:56 ` Paul Mackerras 2007-08-16 7:09 ` Herbert Xu 2007-08-16 8:06 ` Stefan Richter 2007-08-16 8:10 ` Herbert Xu 2007-08-16 9:54 ` Stefan Richter 2007-08-16 10:31 ` Stefan Richter 2007-08-16 10:42 ` Herbert Xu 2007-08-16 16:34 ` Paul E. McKenney 2007-08-16 23:59 ` Herbert Xu 2007-08-17 1:01 ` Paul E. McKenney 2007-08-17 7:39 ` Satyam Sharma 2007-08-17 14:31 ` Paul E. McKenney 2007-08-17 18:31 ` Satyam Sharma 2007-08-17 18:56 ` Paul E. McKenney 2007-08-17 3:15 ` Nick Piggin 2007-08-17 4:02 ` Paul Mackerras 2007-08-17 4:39 ` Nick Piggin 2007-08-17 7:25 ` Stefan Richter 2007-08-17 8:06 ` Nick Piggin 2007-08-17 8:58 ` Satyam Sharma 2007-08-17 9:15 ` Nick Piggin 2007-08-17 10:03 ` Satyam Sharma 2007-08-17 11:50 ` Nick Piggin 2007-08-17 12:50 ` Satyam Sharma 2007-08-17 12:56 ` Nick Piggin 2007-08-17 12:56 ` Nick Piggin 2007-08-18 2:15 ` Satyam Sharma 2007-08-17 10:48 ` Stefan Richter 2007-08-17 10:58 ` Stefan Richter 2007-08-18 14:35 ` LDD3 pitfalls (was Re: [PATCH 0/24] make atomic_read() behave consistently across all architectures) Stefan Richter 2007-08-20 13:28 ` Chris Snook 2007-08-17 22:14 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Segher Boessenkool 2007-08-17 5:04 ` Paul Mackerras 2007-08-16 10:35 ` Herbert Xu 2007-08-16 19:48 ` Chris Snook 2007-08-17 0:02 ` Herbert Xu 2007-08-17 2:04 ` Chris Snook 2007-08-17 2:13 ` Herbert Xu 2007-08-17 2:31 ` Nick Piggin 2007-08-17 5:09 ` Paul Mackerras 2007-08-17 5:32 ` Herbert Xu 2007-08-17 5:41 ` Paul Mackerras 2007-08-17 8:28 ` Satyam Sharma 2007-08-16 14:48 ` Ilpo Järvinen 2007-08-16 16:19 ` Stefan Richter 2007-08-16 19:55 ` Chris Snook 2007-08-16 20:20 ` Christoph Lameter 2007-08-17 1:02 ` Paul E. McKenney 2007-08-17 1:28 ` Herbert Xu 2007-08-17 5:07 ` Paul E. McKenney 2007-08-17 2:16 ` Paul Mackerras 2007-08-17 3:03 ` Linus Torvalds 2007-08-17 3:43 ` Paul Mackerras 2007-08-17 3:53 ` Herbert Xu 2007-08-17 6:26 ` Satyam Sharma 2007-08-17 8:38 ` Nick Piggin 2007-08-17 9:14 ` Satyam Sharma 2007-08-17 9:31 ` Nick Piggin 2007-08-17 10:55 ` Satyam Sharma 2007-08-17 12:39 ` Nick Piggin 2007-08-17 12:39 ` Nick Piggin 2007-08-17 13:36 ` Satyam Sharma 2007-08-17 16:48 ` Linus Torvalds 2007-08-17 18:50 ` Chris Friesen 2007-08-17 18:50 ` Chris Friesen 2007-08-17 18:54 ` Arjan van de Ven 2007-08-17 19:49 ` Paul E. McKenney 2007-08-17 19:49 ` Arjan van de Ven 2007-08-17 20:12 ` Paul E. McKenney 2007-08-17 19:08 ` Linus Torvalds 2007-08-20 13:15 ` Chris Snook 2007-08-20 13:15 ` Chris Snook 2007-08-20 13:32 ` Herbert Xu 2007-08-20 13:38 ` Chris Snook 2007-08-20 22:07 ` Segher Boessenkool 2007-08-21 5:46 ` Linus Torvalds 2007-08-21 7:04 ` David Miller 2007-08-21 13:50 ` Chris Snook 2007-08-21 14:59 ` Segher Boessenkool 2007-08-21 16:31 ` Satyam Sharma 2007-08-21 16:43 ` Linus Torvalds 2007-09-09 18:02 ` Denys Vlasenko 2007-09-09 18:18 ` Arjan van de Ven 2007-09-10 10:56 ` Denys Vlasenko 2007-09-10 11:15 ` Herbert Xu 2007-09-10 12:22 ` Kyle Moffett 2007-09-10 12:22 ` Kyle Moffett 2007-09-10 13:38 ` Denys Vlasenko 2007-09-10 13:38 ` Denys Vlasenko 2007-09-10 14:16 ` Denys Vlasenko 2007-09-10 14:16 ` Denys Vlasenko 2007-09-10 15:09 ` Linus Torvalds 2007-09-10 16:46 ` Denys Vlasenko 2007-09-10 19:59 ` Kyle Moffett 2007-09-10 19:59 ` Kyle Moffett 2007-09-10 18:59 ` Christoph Lameter 2007-09-10 23:19 ` [PATCH] Document non-semantics of atomic_read() and atomic_set() Chris Snook 2007-09-10 23:19 ` Chris Snook 2007-09-10 23:44 ` Paul E. McKenney 2007-09-10 23:44 ` Paul E. McKenney 2007-09-11 19:35 ` Christoph Lameter 2007-09-11 19:35 ` Christoph Lameter 2007-09-10 14:51 ` [PATCH 0/24] make atomic_read() behave consistently across all architectures Arjan van de Ven 2007-09-10 14:38 ` Denys Vlasenko 2007-09-10 17:02 ` Arjan van de Ven 2007-08-17 11:08 ` Stefan Richter 2007-08-17 22:09 ` Segher Boessenkool 2007-08-17 22:09 ` Segher Boessenkool 2007-08-17 17:41 ` Segher Boessenkool 2007-08-17 18:38 ` Satyam Sharma 2007-08-17 23:17 ` Segher Boessenkool 2007-08-17 23:17 ` Segher Boessenkool 2007-08-17 23:55 ` Satyam Sharma 2007-08-18 0:04 ` Segher Boessenkool 2007-08-18 0:04 ` Segher Boessenkool 2007-08-18 1:56 ` Satyam Sharma 2007-08-18 2:15 ` Segher Boessenkool 2007-08-18 2:15 ` Segher Boessenkool 2007-08-18 3:33 ` Satyam Sharma 2007-08-18 5:18 ` Segher Boessenkool 2007-08-18 5:18 ` Segher Boessenkool 2007-08-18 13:20 ` Satyam Sharma 2007-09-10 18:59 ` Christoph Lameter 2007-09-10 20:54 ` Paul E. McKenney 2007-09-10 21:36 ` Christoph Lameter 2007-09-10 21:50 ` Paul E. McKenney 2007-09-11 2:27 ` Segher Boessenkool 2007-09-11 2:27 ` Segher Boessenkool 2007-08-16 21:08 ` Luck, Tony 2007-08-16 21:08 ` Luck, Tony 2007-08-16 19:55 ` Chris Snook 2007-08-16 18:54 ` Christoph Lameter 2007-08-16 20:07 ` Paul E. McKenney 2007-08-16 3:05 ` Paul Mackerras 2007-08-16 19:39 ` Segher Boessenkool 2007-08-16 2:07 ` Segher Boessenkool 2007-08-24 12:50 ` Denys Vlasenko 2007-08-24 17:15 ` Christoph Lameter 2007-08-24 20:21 ` Denys Vlasenko 2007-08-16 3:37 ` Bill Fink 2007-08-16 5:20 ` Satyam Sharma 2007-08-16 5:57 ` Satyam Sharma 2007-08-16 9:25 ` Satyam Sharma 2007-08-16 21:00 ` Segher Boessenkool 2007-08-17 4:32 ` Satyam Sharma 2007-08-17 22:38 ` Segher Boessenkool 2007-08-18 14:42 ` Satyam Sharma 2007-08-16 20:50 ` Segher Boessenkool 2007-08-16 22:40 ` David Schwartz 2007-08-17 4:36 ` Satyam Sharma 2007-08-17 4:24 ` Satyam Sharma 2007-08-17 22:34 ` Segher Boessenkool 2007-08-15 19:59 ` Christoph Lameter
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.