* 3.18.3 + latest bcache-dev
@ 2015-01-21 1:24 Stephen R. van den Berg
2015-01-21 8:00 ` Kent Overstreet
0 siblings, 1 reply; 8+ messages in thread
From: Stephen R. van den Berg @ 2015-01-21 1:24 UTC (permalink / raw)
To: linux-bcache
I have:
16GB RAM
16GB SSD swap
5 x 6TB HDD bcache backing storage
2 x 490GB SSD bcache cache (Crucial M550, 4KB block, 2M bucket size)
writeback and trim enabled on bcache.
I used the latestbcache tools from git.
On the backing storage, I created a BTRFS using nossd and notrim.
Now, whenever I boot, I end up with exactly 4 bcache_writebac
processes pegged at 99 or 100% CPU. This is on an octocore cpu.
top shows it as 50% system time. The four processes are marked as R,
but I cannot trace them.
Even after waiting for several minutes, the cpu load does not come down.
The mounted volume simply seems to work, though.
Any ideas?
--
Stephen.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 3.18.3 + latest bcache-dev
2015-01-21 1:24 3.18.3 + latest bcache-dev Stephen R. van den Berg
@ 2015-01-21 8:00 ` Kent Overstreet
2015-01-21 11:04 ` Stephen R. van den Berg
0 siblings, 1 reply; 8+ messages in thread
From: Kent Overstreet @ 2015-01-21 8:00 UTC (permalink / raw)
To: Stephen R. van den Berg; +Cc: linux-bcache
Try increasing btree_scan_ratelimit?
On Tue, Jan 20, 2015 at 5:24 PM, Stephen R. van den Berg <srb@cuci.nl> wrote:
> I have:
> 16GB RAM
> 16GB SSD swap
> 5 x 6TB HDD bcache backing storage
> 2 x 490GB SSD bcache cache (Crucial M550, 4KB block, 2M bucket size)
> writeback and trim enabled on bcache.
> I used the latestbcache tools from git.
>
> On the backing storage, I created a BTRFS using nossd and notrim.
>
> Now, whenever I boot, I end up with exactly 4 bcache_writebac
> processes pegged at 99 or 100% CPU. This is on an octocore cpu.
> top shows it as 50% system time. The four processes are marked as R,
> but I cannot trace them.
>
> Even after waiting for several minutes, the cpu load does not come down.
> The mounted volume simply seems to work, though.
>
> Any ideas?
> --
> Stephen.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 3.18.3 + latest bcache-dev
2015-01-21 8:00 ` Kent Overstreet
@ 2015-01-21 11:04 ` Stephen R. van den Berg
2015-01-21 16:45 ` Stephen R. van den Berg
2015-01-21 22:47 ` 3.18.3 + latest bcache-dev Slava Pestov
0 siblings, 2 replies; 8+ messages in thread
From: Stephen R. van den Berg @ 2015-01-21 11:04 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcache
Kent Overstreet wrote:
>Try increasing btree_scan_ratelimit?
Well, before I could try this, the 100% CPU usage was gone (after several hours).
However, something else happened:
a. During some heavy traffic mostly writing to the btrfs filesystem
the kernel halted. AFAICS inside some btrfs recovery code.
b. Kernel messages show something like this leading up to the freeze:
Jan 21 11:21:40 ip144 kernel: ata11.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0
Jan 21 11:21:40 ip144 kernel: ata11.00: irq_stat 0x40000008
Jan 21 11:21:40 ip144 kernel: ata11.00: failed command: READ FPDMA QUEUED
Jan 21 11:21:40 ip144 kernel: ata11.00: cmd 60/00:c0:10:d0:d9/04:00:01:00:00/40 tag 24 ncq 524288 in
Jan 21 11:21:40 ip144 kernel: res 41/40:00:d0:d1:d9/00:00:01:00:00/00 Emask 0x409 (media error) <F>
Jan 21 11:21:40 ip144 kernel: ata11.00: status: { DRDY ERR }
Jan 21 11:21:40 ip144 kernel: ata11.00: error: { UNC }
Jan 21 11:21:40 ip144 kernel: ata11.00: configured for UDMA/133
Jan 21 11:21:40 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 11:21:40 ip144 kernel: Result: hostbyte=0x00 driverbyte=0x08
Jan 21 11:21:40 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 11:21:40 ip144 kernel: Sense Key : 0x3 [current] [descriptor]
Jan 21 11:21:40 ip144 kernel: Descriptor sense data with sense descriptors (in hex):
Jan 21 11:21:40 ip144 kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Jan 21 11:21:40 ip144 kernel: 01 d9 d1 d0
Jan 21 11:21:40 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 11:21:40 ip144 kernel: ASC=0x11 ASCQ=0x4
Jan 21 11:21:40 ip144 kernel: sd 10:0:0:0: [sdg] CDB:
Jan 21 11:21:40 ip144 kernel: cdb[0]=0x88: 88 00 00 00 00 00 01 d9 d0 10 00 00 04 00 00 00
Jan 21 11:21:40 ip144 kernel: blk_update_request: I/O error, dev sdg, sector 31052240
Jan 21 11:21:40 ip144 kernel: ata11: EH complete
Jan 21 11:21:40 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
Jan 21 11:21:40 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Jan 21 11:21:40 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
Jan 21 11:21:40 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 4, flush 0, corrupt 0, gen 0
Jan 21 11:21:40 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 5, flush 0, corrupt 0, gen 0
The above repeats like 10 times in 4 second intervals (more or less), and then a final:
Jan 21 11:35:32 ip144 kernel: ata11.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0
Jan 21 11:35:32 ip144 kernel: ata11.00: irq_stat 0x40000008
Jan 21 11:35:32 ip144 kernel: ata11.00: failed command: READ FPDMA QUEUED
Jan 21 11:35:32 ip144 kernel: ata11.00: cmd 60/80:58:90:f2:da/01:00:01:00:00/40 tag 11 ncq 196608 in
Jan 21 11:35:32 ip144 kernel: res 41/40:00:90:f2:da/00:00:01:00:00/00 Emask 0x409 (media error) <F>
Jan 21 11:35:32 ip144 kernel: ata11.00: status: { DRDY ERR }
Jan 21 11:35:32 ip144 kernel: ata11.00: error: { UNC }
Jan 21 11:35:32 ip144 kernel: ata11.00: configured for UDMA/133
Jan 21 11:35:32 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 11:35:32 ip144 kernel: Result: hostbyte=0x00 driverbyte=0x08
Jan 21 11:35:32 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 11:35:32 ip144 kernel: Sense Key : 0x3 [current] [descriptor]
Jan 21 11:35:32 ip144 kernel: Descriptor sense data with sense descriptors (in hex):
Jan 21 11:35:32 ip144 kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Jan 21 11:35:32 ip144 kernel: 01 da f2 90
Jan 21 11:35:32 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 11:35:32 ip144 kernel: ASC=0x11 ASCQ=0x4
Jan 21 11:35:32 ip144 kernel: sd 10:0:0:0: [sdg] CDB:
Jan 21 11:35:32 ip144 kernel: cdb[0]=0x88: 88 00 00 00 00 00 01 da f2 90 00 00 01 80 00 00
Jan 21 11:35:32 ip144 kernel: blk_update_request: I/O error, dev sdg, sector 31126160
Jan 21 11:35:32 ip144 kernel: ata11: EH complete
Jan 21 11:35:32 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 130, flush 0, corrupt 0, gen 0
Jan 21 11:35:32 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 131, flush 0, corrupt 0, gen 0
Jan 21 11:35:32 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 132, flush 0, corrupt 0, gen 0
Jan 21 11:35:32 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 133, flush 0, corrupt 0, gen 0
Jan 21 11:35:32 ip144 kernel: BTRFS: bdev /dev/bcache2 errs: wr 0, rd 134, flush 0, corrupt 0, gen 0
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131932160 (dev /dev/bcache2 sector 31126160)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131981312 (dev /dev/bcache2 sector 31126256)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131936256 (dev /dev/bcache2 sector 31126168)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131964928 (dev /dev/bcache2 sector 31126224)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131923968 (dev /dev/bcache2 sector 31126144)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131948544 (dev /dev/bcache2 sector 31126192)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131944448 (dev /dev/bcache2 sector 31126184)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 131956736 (dev /dev/bcache2 sector 31126208)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 132001792 (dev /dev/bcache2 sector 31126296)
Jan 21 11:35:33 ip144 kernel: BTRFS: read error corrected: ino 318562 off 132009984 (dev /dev/bcache2 sector 31126312)
After that the system freezes with a kernel oops in btrfs land.
c. Upon reboot, the kernel reports the following:
Jan 21 11:43:46 ip144 kernel: bcache: register_cache() registered cache device sda3
Jan 21 11:43:46 ip144 kernel: bcache: register_cache() registered cache device sdb3
Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdg
Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdc
Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdf
Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sde
Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdd
Jan 21 11:43:46 ip144 kernel: bcache: error on 04063132-f7e1-4f29-a3aa-675491b426f4: sdb3: bad magic (got 17495323148306469098 expect 5629218783827753365) while reading prios from bucket 16868
Jan 21 11:43:46 ip144 kernel: bcache: register_bcache() error opening /dev/sdb3: error reading priorities
Jan 21 11:43:46 ip144 kernel: bcache: bch_cache_read_only() sda3 read only
Jan 21 11:43:46 ip144 kernel: bcache: bch_cache_read_only() sdb3 read only
Jan 21 11:43:46 ip144 kernel: bcache: cache_set_free() Cache set 04063132-f7e1-4f29-a3aa-675491b426f4 unregistered
Jan 21 11:43:46 ip144 kernel: bcache: bch_cache_release() sda3 removed
Jan 21 11:43:46 ip144 kernel: bcache: bch_cache_release() sdb3 removed
Now, the data on this filesystem is non-critical, so I could either attempt a repair
or wipe and start over. Any suggestions?
Writeback was on, so simply reformatting sdb3 most likely will result in mayhem.
I should probably take out sdg and replace it with a drive without errors.
--
Stephen.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 3.18.3 + latest bcache-dev
2015-01-21 11:04 ` Stephen R. van den Berg
@ 2015-01-21 16:45 ` Stephen R. van den Berg
2015-01-22 12:33 ` patch (Re: 3.18.3 + latest bcache-dev) Stephen R. van den Berg
2015-01-21 22:47 ` 3.18.3 + latest bcache-dev Slava Pestov
1 sibling, 1 reply; 8+ messages in thread
From: Stephen R. van den Berg @ 2015-01-21 16:45 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcache
Well, what I did was:
- Take out sdg.
- Reformat both caching and backing devices and started over.
I still have the kernel logs of the previous fault condition, but
not the data anymore.
What I notice are:
- The 100% CPU usage for four bcache_writebac processes is back. I've
already tried raising the value you suggested from 30 to 60, but I did
not have an opportunity to reboot yet. Will report back on this.
- When I run bcache-super-show on the backing devices, it works just fine.
- When I run bcache-super-show on the caching devices (while in service)
I get something like this:
sb.magic ok
sb.first_sector 8 [match]
sb.csum FFFFFFFF5D55BF8D [expected 726D3B42D08769EC]
Corrupt superblock (bad csum)
On both caching devices. Is this to be expected?
--
Stephen.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 3.18.3 + latest bcache-dev
2015-01-21 11:04 ` Stephen R. van den Berg
2015-01-21 16:45 ` Stephen R. van den Berg
@ 2015-01-21 22:47 ` Slava Pestov
2015-01-21 23:04 ` Stephen R. van den Berg
1 sibling, 1 reply; 8+ messages in thread
From: Slava Pestov @ 2015-01-21 22:47 UTC (permalink / raw)
To: Stephen R. van den Berg; +Cc: Kent Overstreet, linux-bcache
On Wed, Jan 21, 2015 at 3:04 AM, Stephen R. van den Berg <srb@cuci.nl> wrote:
> Jan 21 11:21:40 ip144 kernel: ata11.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0
> Jan 21 11:21:40 ip144 kernel: ata11.00: irq_stat 0x40000008
> Jan 21 11:21:40 ip144 kernel: ata11.00: failed command: READ FPDMA QUEUED
> Jan 21 11:21:40 ip144 kernel: ata11.00: cmd 60/00:c0:10:d0:d9/04:00:01:00:00/40 tag 24 ncq 524288 in
> Jan 21 11:21:40 ip144 kernel: res 41/40:00:d0:d1:d9/00:00:01:00:00/00 Emask 0x409 (media error) <F>
> Jan 21 11:21:40 ip144 kernel: ata11.00: status: { DRDY ERR }
> Jan 21 11:21:40 ip144 kernel: ata11.00: error: { UNC }
> Jan 21 11:21:40 ip144 kernel: ata11.00: configured for UDMA/133
>
I'm not sure this is related. Do you see it during normal operation
ever? It is possible that we're spinning in softirq context or
something, starving the device, but I'm not sure...
> Jan 21 11:43:46 ip144 kernel: bcache: register_cache() registered cache device sda3
> Jan 21 11:43:46 ip144 kernel: bcache: register_cache() registered cache device sdb3
> Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdg
> Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdc
> Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdf
> Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sde
> Jan 21 11:43:46 ip144 kernel: bcache: register_bdev() registered backing device sdd
> Jan 21 11:43:46 ip144 kernel: bcache: error on 04063132-f7e1-4f29-a3aa-675491b426f4: sdb3: bad magic (got 17495323148306469098 expect 5629218783827753365) while reading prios from bucket 16868
> Jan 21 11:43:46 ip144 kernel: bcache: register_bcache() error opening /dev/sdb3: error reading priorities
This is a bcache-dev bug that we just hit in our automation runs after
adding some more reboot tests. We're working on it.
Slava
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: 3.18.3 + latest bcache-dev
2015-01-21 22:47 ` 3.18.3 + latest bcache-dev Slava Pestov
@ 2015-01-21 23:04 ` Stephen R. van den Berg
0 siblings, 0 replies; 8+ messages in thread
From: Stephen R. van den Berg @ 2015-01-21 23:04 UTC (permalink / raw)
To: Slava Pestov; +Cc: Kent Overstreet, linux-bcache
Slava Pestov wrote:
>On Wed, Jan 21, 2015 at 3:04 AM, Stephen R. van den Berg <srb@cuci.nl> wrote:
>> Jan 21 11:21:40 ip144 kernel: ata11.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0
>> Jan 21 11:21:40 ip144 kernel: ata11.00: irq_stat 0x40000008
>> Jan 21 11:21:40 ip144 kernel: ata11.00: failed command: READ FPDMA QUEUED
>> Jan 21 11:21:40 ip144 kernel: ata11.00: cmd 60/00:c0:10:d0:d9/04:00:01:00:00/40 tag 24 ncq 524288 in
>> Jan 21 11:21:40 ip144 kernel: res 41/40:00:d0:d1:d9/00:00:01:00:00/00 Emask 0x409 (media error) <F>
>> Jan 21 11:21:40 ip144 kernel: ata11.00: status: { DRDY ERR }
>> Jan 21 11:21:40 ip144 kernel: ata11.00: error: { UNC }
>> Jan 21 11:21:40 ip144 kernel: ata11.00: configured for UDMA/133
>I'm not sure this is related. Do you see it during normal operation
>ever? It is possible that we're spinning in softirq context or
>something, starving the device, but I'm not sure...
In the second run now, with increased traffic, I do not see the above
happening.
Maybe I cut it short, the whole message is:
Jan 21 12:09:14 ip144 kernel: ata11.00: exception Emask 0x0 SAct 0x3000 SErr 0x0 action 0x0
Jan 21 12:09:14 ip144 kernel: ata11.00: irq_stat 0x40000008
Jan 21 12:09:14 ip144 kernel: ata11.00: failed command: READ FPDMA QUEUED
Jan 21 12:09:14 ip144 kernel: ata11.00: cmd 60/00:68:e8:2b:da/01:00:01:00:00/40 tag 13 ncq 131072 in
Jan 21 12:09:14 ip144 kernel: res 41/40:00:b0:2c:da/00:00:01:00:00/00 Emask 0x409 (media error) <F>
Jan 21 12:09:14 ip144 kernel: ata11.00: status: { DRDY ERR }
Jan 21 12:09:14 ip144 kernel: ata11.00: error: { UNC }
Jan 21 12:09:14 ip144 kernel: ata11.00: configured for UDMA/133
Jan 21 12:09:14 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 12:09:14 ip144 kernel: Result: hostbyte=0x00 driverbyte=0x08
Jan 21 12:09:14 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 12:09:14 ip144 kernel: Sense Key : 0x3 [current] [descriptor]
Jan 21 12:09:14 ip144 kernel: Descriptor sense data with sense descriptors (in hex):
Jan 21 12:09:14 ip144 kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Jan 21 12:09:14 ip144 kernel: 01 da 2c b0
Jan 21 12:09:14 ip144 kernel: sd 10:0:0:0: [sdg]
Jan 21 12:09:14 ip144 kernel: ASC=0x11 ASCQ=0x4
Jan 21 12:09:14 ip144 kernel: sd 10:0:0:0: [sdg] CDB:
Jan 21 12:09:14 ip144 kernel: cdb[0]=0x88: 88 00 00 00 00 00 01 da 2b e8 00 00 01 00 00 00
Jan 21 12:09:14 ip144 kernel: blk_update_request: I/O error, dev sdg, sector 31075504
Jan 21 12:09:14 ip144 kernel: ata11: EH complete
And it always happens upon a read error from HDD.
--
Stephen.
^ permalink raw reply [flat|nested] 8+ messages in thread
* patch (Re: 3.18.3 + latest bcache-dev)
2015-01-21 16:45 ` Stephen R. van den Berg
@ 2015-01-22 12:33 ` Stephen R. van den Berg
2015-01-22 12:56 ` Stephen R. van den Berg
0 siblings, 1 reply; 8+ messages in thread
From: Stephen R. van den Berg @ 2015-01-22 12:33 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcache
Stephen R. van den Berg wrote:
>What I notice are:
>- The 100% CPU usage for four bcache_writebac processes is back. I've
> already tried raising the value you suggested from 30 to 60, but I did
> not have an opportunity to reboot yet. Will report back on this.
What I saw:
a. After running several hours, copying roughly 3.0TB through the two
SSD caching devices onto the BTRFS backing device consisting of
four HDDs, the bcache_writebac processes were still eating 100% CPU each.
b. Then did a manual umount of the fs, then stopped the bcache device
through the proc filesystem.
c. Rebooted.
d. Came back in a patched kernel, 100% CPU usage is gone.
The patch I used (I did not investigate the correctness of the value
picked, I simply doubled it as an initial guess):
commit 2abecdbd214c94741012de32759bb0de4c6adbf9
Author: Stephen R. van den Berg <srb@cuci.nl>
Date: Tue Jan 13 23:24:52 2015 +0100
Increase btree_scan_ratelimit to avoid 100% CPU hogging.
diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
index 653419b..5fff8ad 100644
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -867,7 +867,7 @@ static struct cache_set *bch_cache_set_alloc(struct cache *ca)
c->error_limit = 16 << IO_ERROR_SHIFT;
c->tiering_percent = 10;
- c->btree_scan_ratelimit = 30;
+ c->btree_scan_ratelimit = 60;
c->copy_gc_enabled = 1;
c->tiering_enabled = 1;
--
Stephen.
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: patch (Re: 3.18.3 + latest bcache-dev)
2015-01-22 12:33 ` patch (Re: 3.18.3 + latest bcache-dev) Stephen R. van den Berg
@ 2015-01-22 12:56 ` Stephen R. van den Berg
0 siblings, 0 replies; 8+ messages in thread
From: Stephen R. van den Berg @ 2015-01-22 12:56 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcache
Stephen R. van den Berg wrote:
>d. Came back in a patched kernel, 100% CPU usage is gone.
I spoke too soon. After some more copying, the 100% CPU is back, but
different:
Before I had four or five HDDs, but in both cases I had four bcache_writebac
processes running at 100%.
Now I have four HDDs, and also four bcache_writebac processes, but
only three of those run at 100% CPU. The fourth one is idle (mostly).
--
Stephen.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-01-22 12:56 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-21 1:24 3.18.3 + latest bcache-dev Stephen R. van den Berg
2015-01-21 8:00 ` Kent Overstreet
2015-01-21 11:04 ` Stephen R. van den Berg
2015-01-21 16:45 ` Stephen R. van den Berg
2015-01-22 12:33 ` patch (Re: 3.18.3 + latest bcache-dev) Stephen R. van den Berg
2015-01-22 12:56 ` Stephen R. van den Berg
2015-01-21 22:47 ` 3.18.3 + latest bcache-dev Slava Pestov
2015-01-21 23:04 ` Stephen R. van den Berg
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.