dm-cache fs corruption

dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 844 bytes --]

Hello,
I've been testing dm-cache for use with static web content.
It appears that, when using big cache ~3TB(+20TB origin dev) dm-cache
device corrupts the filesystem after block eviction starts to happen.
If I set smaller cache size - single 480GB ssd(again with 20TB origin dev),
the dm-cache dev works just fine.


dmesg error:

XFS (dm-0): Corruption detected. Unmount and run xfs_repair
XFS (dm-0): metadata I/O error: block 0x3d4f534e0
("xfs_trans_read_buf_map") error 117 numblks 16
XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.
XFS (dm-0): Corruption detected. Unmount and run xfs_repair
XFS (dm-0): Corruption detected. Unmount and run xfs_repair


ext4 also got broken.


Tested with 3.10.11-gentoo, fedora 19- kernel 3.11.something, xfs, ext4,
writeback and writethrough modes.



regards,
Vladimir Smolensky

[-- Attachment #1.2: Type: text/html, Size: 1073 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Joe Thornber]

On Wed, Nov 13, 2013 at 12:24:03PM +0200, Vladimir Smolensky wrote:
> Hello,
> I've been testing dm-cache for use with static web content.
> It appears that, when using big cache ~3TB(+20TB origin dev) dm-cache
> device corrupts the filesystem after block eviction starts to happen.
> If I set smaller cache size - single 480GB ssd(again with 20TB origin dev),
> the dm-cache dev works just fine.

Hi Vladmimir,

Sorry you've been having trouble, and thanks for taking the time to
report.  I'll see if I can scrape together enough disks to reproduce
the problem, and get back to you.

- Joe

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 961 bytes --]

Hello Joe,
I can provide you with the setup to test if you want.
Problem is, 3TB cache gets filled very slowly - about 2 weeks.

regards,
Vladimir Smolensky


On Wed, Nov 13, 2013 at 3:10 PM, Joe Thornber <thornber@redhat.com> wrote:

> On Wed, Nov 13, 2013 at 12:24:03PM +0200, Vladimir Smolensky wrote:
> > Hello,
> > I've been testing dm-cache for use with static web content.
> > It appears that, when using big cache ~3TB(+20TB origin dev) dm-cache
> > device corrupts the filesystem after block eviction starts to happen.
> > If I set smaller cache size - single 480GB ssd(again with 20TB origin
> dev),
> > the dm-cache dev works just fine.
>
> Hi Vladmimir,
>
> Sorry you've been having trouble, and thanks for taking the time to
> report.  I'll see if I can scrape together enough disks to reproduce
> the problem, and get back to you.
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 1564 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Mike Snitzer]

On Wed, Nov 13 2013 at  8:48am -0500,
Vladimir Smolensky <arizal@gmail.com> wrote:

> Hello Joe,
> I can provide you with the setup to test if you want.
> Problem is, 3TB cache gets filled very slowly - about 2 weeks.

We'll have to take steps to get a quicker reproducer.  But in parallel,
could you please test the latest dm-cache code that is staged for v3.13
inclusion?  Would be a great data point to have.

Please see the 'for-linus' branch of the linux-dm.git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git

(it is based on 3.12-rc5)

Re: dm-cache fs corruption

[Joe Thornber]

On Wed, Nov 13, 2013 at 03:48:18PM +0200, Vladimir Smolensky wrote:
> Hello Joe,
> I can provide you with the setup to test if you want.
> Problem is, 3TB cache gets filled very slowly - about 2 weeks.

I'll use cache_restore to generate a populated cache.  Could you tell
me your block size though please?

- Joe

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 520 bytes --]

8192



On Wed, Nov 13, 2013 at 4:41 PM, Joe Thornber <thornber@redhat.com> wrote:

> On Wed, Nov 13, 2013 at 03:48:18PM +0200, Vladimir Smolensky wrote:
> > Hello Joe,
> > I can provide you with the setup to test if you want.
> > Problem is, 3TB cache gets filled very slowly - about 2 weeks.
>
> I'll use cache_restore to generate a populated cache.  Could you tell
> me your block size though please?
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 1133 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 156 bytes --]

Hello,
I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok right?

Where can I get cache_restore, cache_dump, etc. and info how to use them.

[-- Attachment #1.2: Type: text/html, Size: 241 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Mears, Morgan]

On Wed, Nov 13, 2013 at 11:29:26AM -0005, Vladimir Smolensky wrote:
> Hello, 
> I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok right?
> 
> Where can I get cache_restore, cache_dump, etc. and info how to use them.

Hi Vladimir,

git clone https://github.com/jthornber/thin-provisioning-tools.git
cd thin-provisioning-tools
autoreconf
./configure --enable-testing
make
sudo cp cache_check cache_dump cache_restore cache_repair /usr/sbin
cache_check --version

This is mostly distilled from README.md in the repo; there are some
additional dependencies in there that you might have to address.

All the tools have a --help option; not sure about other documentation.

--Morgan

Re: dm-cache fs corruption

[Mike Snitzer]

On Wed, Nov 13 2013 at 11:29am -0500,
Vladimir Smolensky <arizal@gmail.com> wrote:

> Hello,
> I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok right?

Yes, that branch is based on v3.12-rc5.  Once the branch is made it is
best not to rebase (Linus prefers that, unless there is a strong reason
to rebase).

Chances are there is still a 64bit bug that is causing truncation of
values to 32bit.  So I wouldn't expect the latest code to help your
particular situation but it doesn't hurt to use the latest.

> Where can I get cache_restore, cache_dump, etc. and info how to use them.

Those tools are part of this project:
https://github.com/jthornber/thin-provisioning-tools

There are man pages for each utility (in the man8 subdir).

Re: dm-cache fs corruption

[Joe Thornber]

On Wed, Nov 13, 2013 at 06:39:37PM +0000, Mears, Morgan wrote:
> On Wed, Nov 13, 2013 at 11:29:26AM -0005, Vladimir Smolensky wrote:
> > Hello, 
> > I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok right?
> > 
> > Where can I get cache_restore, cache_dump, etc. and info how to use them.
> 
> Hi Vladimir,
> 
> git clone https://github.com/jthornber/thin-provisioning-tools.git
> cd thin-provisioning-tools
> autoreconf
> ./configure --enable-testing
> make
> sudo cp cache_check cache_dump cache_restore cache_repair /usr/sbin
> cache_check --version
> 
> This is mostly distilled from README.md in the repo; there are some
> additional dependencies in there that you might have to address.
> 
> All the tools have a --help option; not sure about other documentation.
> 
> --Morgan

'make install' should work, and I believe there are some man pages.

- Joe

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 1135 bytes --]

./cache_restore -V
0.2.8


On Thu, Nov 14, 2013 at 4:38 PM, Joe Thornber <thornber@redhat.com> wrote:

> On Wed, Nov 13, 2013 at 06:39:37PM +0000, Mears, Morgan wrote:
> > On Wed, Nov 13, 2013 at 11:29:26AM -0005, Vladimir Smolensky wrote:
> > > Hello,
> > > I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok
> right?
> > >
> > > Where can I get cache_restore, cache_dump, etc. and info how to use
> them.
> >
> > Hi Vladimir,
> >
> > git clone https://github.com/jthornber/thin-provisioning-tools.git
> > cd thin-provisioning-tools
> > autoreconf
> > ./configure --enable-testing
> > make
> > sudo cp cache_check cache_dump cache_restore cache_repair /usr/sbin
> > cache_check --version
> >
> > This is mostly distilled from README.md in the repo; there are some
> > additional dependencies in there that you might have to address.
> >
> > All the tools have a --help option; not sure about other documentation.
> >
> > --Morgan
>
> 'make install' should work, and I believe there are some man pages.
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 1962 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 1552 bytes --]

Okay, a little correction here. The problematic setup was with 2.3TB cache,
not 3TB. The mistake comes from the fact that at the time of the writing I
was dealing with another server which has 8x480GB ssd's in raid5, which is
little over 3TB. Obviously the number was floating in my head.


On Thu, Nov 14, 2013 at 5:21 PM, Vladimir Smolensky <arizal@gmail.com>wrote:

> ./cache_restore -V
> 0.2.8
>
>
> On Thu, Nov 14, 2013 at 4:38 PM, Joe Thornber <thornber@redhat.com> wrote:
>
>> On Wed, Nov 13, 2013 at 06:39:37PM +0000, Mears, Morgan wrote:
>> > On Wed, Nov 13, 2013 at 11:29:26AM -0005, Vladimir Smolensky wrote:
>> > > Hello,
>> > > I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok
>> right?
>> > >
>> > > Where can I get cache_restore, cache_dump, etc. and info how to use
>> them.
>> >
>> > Hi Vladimir,
>> >
>> > git clone https://github.com/jthornber/thin-provisioning-tools.git
>> > cd thin-provisioning-tools
>> > autoreconf
>> > ./configure --enable-testing
>> > make
>> > sudo cp cache_check cache_dump cache_restore cache_repair /usr/sbin
>> > cache_check --version
>> >
>> > This is mostly distilled from README.md in the repo; there are some
>> > additional dependencies in there that you might have to address.
>> >
>> > All the tools have a --help option; not sure about other documentation.
>> >
>> > --Morgan
>>
>> 'make install' should work, and I believe there are some man pages.
>>
>> - Joe
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>
>

[-- Attachment #1.2: Type: text/html, Size: 2617 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 3839 bytes --]

Okay, I'm trying to reproduce the problem using cache_restore, so far with
no luck. The hardware setup is the same as the original.

kernel 3.10.11-gentoo


Cache raid, 1 is meta, 2 data;

# parted /dev/sdc unit b print
Model: DELL PERC H700 (scsi)
Disk /dev/sdc: 2397799710720B
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start        End             Size            File system  Name
Flags
 1      1048576B     1024458751B     1023410176B                  primary
 2      1024458752B  2397798662143B  2396774203392B               primary

Origin:

# parted /dev/sdb unit b print
Model: DELL PERC H700 (scsi)
Disk /dev/sdb: 20973392756736B
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start     End              Size             File system  Name
Flags
 1      1048576B  20973391708159B  20973390659584B  xfs          primary

According to /dev/sda2 size, number of blocks should be

571435, with block size 4MiB( 8192 sectors)

Generating fake meta, I leave 5 blocks free

cache_xml create --nr-cache-blocks 571430 --nr-mappings 571430 0  >
/root/meta.xml

I edit the xml to set block size 8192 since there is no option for this.

cache_restore -i /root/meta.xml -o /dev/sdc1

dmsetup create storage --table '0 40963653632 cache /dev/sdc1 /dev/sdc2
/dev/sdb1 8192 1 writeback default 0’


mkfs.xfs /dev/mapper/storage

mount //dev/mapper/storage /mnt

Then I run fio with following setup:

[test1]
loops=10000
randrepeat=1
directory=/mnt/fio/data/
new_group
group_reporting=1
size=100g
rwmixread=70
rw=randrw
numjobs=12
ioengine=sync
#direct=1
bs=1M
nrfiles=3000

This is the status so far:

# dmsetup status storage
0 40963653632 cache 1719/249856 93161125 11990840 93713813 27533087 307339
307344 571435 306978 0 2 migration_threshold 10000000 4 random_threshold 4
sequential_threshold 10000000

There are ~300k demotions with no crash so far.
So probably filling the cache/demotions are not the only factor here.

regards.




On Thu, Nov 14, 2013 at 10:22 PM, Vladimir Smolensky <arizal@gmail.com>wrote:

> Okay, a little correction here. The problematic setup was with 2.3TB
> cache, not 3TB. The mistake comes from the fact that at the time of the
> writing I was dealing with another server which has 8x480GB ssd's in raid5,
> which is little over 3TB. Obviously the number was floating in my head.
>
>
> On Thu, Nov 14, 2013 at 5:21 PM, Vladimir Smolensky <arizal@gmail.com>wrote:
>
>> ./cache_restore -V
>> 0.2.8
>>
>>
>> On Thu, Nov 14, 2013 at 4:38 PM, Joe Thornber <thornber@redhat.com>wrote:
>>
>>> On Wed, Nov 13, 2013 at 06:39:37PM +0000, Mears, Morgan wrote:
>>> > On Wed, Nov 13, 2013 at 11:29:26AM -0005, Vladimir Smolensky wrote:
>>> > > Hello,
>>> > > I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok
>>> right?
>>> > >
>>> > > Where can I get cache_restore, cache_dump, etc. and info how to use
>>> them.
>>> >
>>> > Hi Vladimir,
>>> >
>>> > git clone https://github.com/jthornber/thin-provisioning-tools.git
>>> > cd thin-provisioning-tools
>>> > autoreconf
>>> > ./configure --enable-testing
>>> > make
>>> > sudo cp cache_check cache_dump cache_restore cache_repair /usr/sbin
>>> > cache_check --version
>>> >
>>> > This is mostly distilled from README.md in the repo; there are some
>>> > additional dependencies in there that you might have to address.
>>> >
>>> > All the tools have a --help option; not sure about other documentation.
>>> >
>>> > --Morgan
>>>
>>> 'make install' should work, and I believe there are some man pages.
>>>
>>> - Joe
>>>
>>> --
>>> dm-devel mailing list
>>> dm-devel@redhat.com
>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>>
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 5660 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 4281 bytes --]

Hello, I would like to test the latest dm-cache available to see if the
problem I was having still exists. Where can I get this code?

regards.



On Fri, Nov 15, 2013 at 3:01 PM, Vladimir Smolensky <arizal@gmail.com>wrote:

> Okay, I'm trying to reproduce the problem using cache_restore, so far with
> no luck. The hardware setup is the same as the original.
>
> kernel 3.10.11-gentoo
>
>
> Cache raid, 1 is meta, 2 data;
>
> # parted /dev/sdc unit b print
> Model: DELL PERC H700 (scsi)
> Disk /dev/sdc: 2397799710720B
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
>
> Number  Start        End             Size            File system  Name
> Flags
>  1      1048576B     1024458751B     1023410176B                  primary
>  2      1024458752B  2397798662143B  2396774203392B               primary
>
> Origin:
>
> # parted /dev/sdb unit b print
> Model: DELL PERC H700 (scsi)
> Disk /dev/sdb: 20973392756736B
> Sector size (logical/physical): 512B/512B
> Partition Table: gpt
> Disk Flags:
>
> Number  Start     End              Size             File system  Name
> Flags
>  1      1048576B  20973391708159B  20973390659584B  xfs          primary
>
> According to /dev/sda2 size, number of blocks should be
>
> 571435, with block size 4MiB( 8192 sectors)
>
> Generating fake meta, I leave 5 blocks free
>
> cache_xml create --nr-cache-blocks 571430 --nr-mappings 571430 0  >
> /root/meta.xml
>
> I edit the xml to set block size 8192 since there is no option for this.
>
> cache_restore -i /root/meta.xml -o /dev/sdc1
>
> dmsetup create storage --table '0 40963653632 cache /dev/sdc1 /dev/sdc2
> /dev/sdb1 8192 1 writeback default 0’
>
>
> mkfs.xfs /dev/mapper/storage
>
> mount //dev/mapper/storage /mnt
>
> Then I run fio with following setup:
>
> [test1]
> loops=10000
> randrepeat=1
> directory=/mnt/fio/data/
> new_group
> group_reporting=1
> size=100g
> rwmixread=70
> rw=randrw
> numjobs=12
> ioengine=sync
> #direct=1
> bs=1M
> nrfiles=3000
>
> This is the status so far:
>
> # dmsetup status storage
> 0 40963653632 cache 1719/249856 93161125 11990840 93713813 27533087 307339
> 307344 571435 306978 0 2 migration_threshold 10000000 4 random_threshold 4
> sequential_threshold 10000000
>
> There are ~300k demotions with no crash so far.
> So probably filling the cache/demotions are not the only factor here.
>
> regards.
>
>
>
>
> On Thu, Nov 14, 2013 at 10:22 PM, Vladimir Smolensky <arizal@gmail.com>wrote:
>
>> Okay, a little correction here. The problematic setup was with 2.3TB
>> cache, not 3TB. The mistake comes from the fact that at the time of the
>> writing I was dealing with another server which has 8x480GB ssd's in raid5,
>> which is little over 3TB. Obviously the number was floating in my head.
>>
>>
>> On Thu, Nov 14, 2013 at 5:21 PM, Vladimir Smolensky <arizal@gmail.com>wrote:
>>
>>> ./cache_restore -V
>>> 0.2.8
>>>
>>>
>>> On Thu, Nov 14, 2013 at 4:38 PM, Joe Thornber <thornber@redhat.com>wrote:
>>>
>>>> On Wed, Nov 13, 2013 at 06:39:37PM +0000, Mears, Morgan wrote:
>>>> > On Wed, Nov 13, 2013 at 11:29:26AM -0005, Vladimir Smolensky wrote:
>>>> > > Hello,
>>>> > > I'm compiling the for-linus branch, it shows 3.12.0-rc5, that's ok
>>>> right?
>>>> > >
>>>> > > Where can I get cache_restore, cache_dump, etc. and info how to use
>>>> them.
>>>> >
>>>> > Hi Vladimir,
>>>> >
>>>> > git clone https://github.com/jthornber/thin-provisioning-tools.git
>>>> > cd thin-provisioning-tools
>>>> > autoreconf
>>>> > ./configure --enable-testing
>>>> > make
>>>> > sudo cp cache_check cache_dump cache_restore cache_repair /usr/sbin
>>>> > cache_check --version
>>>> >
>>>> > This is mostly distilled from README.md in the repo; there are some
>>>> > additional dependencies in there that you might have to address.
>>>> >
>>>> > All the tools have a --help option; not sure about other
>>>> documentation.
>>>> >
>>>> > --Morgan
>>>>
>>>> 'make install' should work, and I believe there are some man pages.
>>>>
>>>> - Joe
>>>>
>>>> --
>>>> dm-devel mailing list
>>>> dm-devel@redhat.com
>>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>>>
>>>
>>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 6241 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Joe Thornber]

On Thu, Nov 28, 2013 at 06:50:39PM +0200, Vladimir Smolensky wrote:
> Hello, I would like to test the latest dm-cache available to see if the
> problem I was having still exists. Where can I get this code?

My latest stable code lives in the 'thin-dev' branch of my github tree:

https://github.com/jthornber/linux-2.6/tree/thin-dev

This has the latest versions of both the thin-provisioning and cache
targets, and will shortly have the era target too.


- Joe

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 699 bytes --]

This is complete kernel, right?


On Thu, Nov 28, 2013 at 7:28 PM, Joe Thornber <thornber@redhat.com> wrote:

> On Thu, Nov 28, 2013 at 06:50:39PM +0200, Vladimir Smolensky wrote:
> > Hello, I would like to test the latest dm-cache available to see if the
> > problem I was having still exists. Where can I get this code?
>
> My latest stable code lives in the 'thin-dev' branch of my github tree:
>
> https://github.com/jthornber/linux-2.6/tree/thin-dev
>
> This has the latest versions of both the thin-provisioning and cache
> targets, and will shortly have the era target too.
>
>
> - Joe
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

[-- Attachment #1.2: Type: text/html, Size: 1347 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 5577 bytes --]

Ok, I compiled it and will test if it works.

When I try to make my cache with blocksize smaller than 8192  I get

# dmsetup  create storage --table '0 40963653632 cache /dev/sdc1 /dev/sdc2
/dev/sdb1 4096 1 writeback default 0'
device-mapper: reload ioctl on storage failed: Cannot allocate memory
Command failed

dnesg:

[  664.940597] device-mapper: cache-policy-mq: version 1.1.0 loaded
[  665.002924] ------------[ cut here ]------------
[  665.002943] WARNING: CPU: 0 PID: 18271 at mm/page_alloc.c:2484
__alloc_pages_nodemask+0x73d/0x820()
[  665.002945] Modules linked in: dm_cache_mq dm_cache dm_bio_prison
dm_persistent_data dm_bufio ipv6 bonding coretemp ixgbe i7core_edac
edac_core kvm processor pcspkr hed button bnx2 dcdbas dca mdio ehci_pci
thermal_sys microcode joydev sha256_generic libiscsi scsi_transport_iscsi
tg3 ptp pps_core libphy e1000 fuse nfs lockd sunrpc jfs multipath linear
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor xor
async_tx raid6_pq raid1 raid0 dm_snapshot dm_crypt dm_mirror dm_region_hash
dm_log dm_mod hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx
hid_gyration sl811_hcd usbhid ohci_hcd uhci_hcd usb_storage ehci_hcd
usbcore usb_common mpt2sas raid_class aic94xx libsas lpfc crc_t10dif
crct10dif_common qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid
aacraid sx8 DAC960 cciss 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc
scsi_transport_fc scsi_tgt mptspi mptscsih mptbase atp870u dc395x qla1280
imm parport dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx
aic79xx scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci
libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via
sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs5530
pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell
pata_sc1200 pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd
pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_artop
pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366
pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x
pata_mpiix libata
[  665.003077] CPU: 0 PID: 18271 Comm: dmsetup Not tainted 3.12.0-rc5+ #1
[  665.003079] Hardware name: Dell Inc. PowerEdge R510/0DPRKF, BIOS 1.5.3
10/25/2010
[  665.003082]  00000000000009b4 ffff881fada458f8 ffffffff81595198
00000000000009b4
[  665.003085]  0000000000000000 ffff881fada45938 ffffffff81043a62
ffff881fada45918
[  665.003088]  0000000000000000 ffff881fae8a0000 0000000000000000
0000000000000002
[  665.003092] Call Trace:
[  665.003101]  [<ffffffff81595198>] dump_stack+0x49/0x61
[  665.003107]  [<ffffffff81043a62>] warn_slowpath_common+0x82/0xb0
[  665.003111]  [<ffffffff81043aa5>] warn_slowpath_null+0x15/0x20
[  665.003114]  [<ffffffff810c595d>] __alloc_pages_nodemask+0x73d/0x820
[  665.003120]  [<ffffffffa14405b7>] ? mq_create+0x187/0x3a0 [dm_cache_mq]
[  665.003124]  [<ffffffff810c5ac2>] __get_free_pages+0x12/0x50
[  665.003130]  [<ffffffff810f69cb>] __kmalloc+0xeb/0xf0
[  665.003134]  [<ffffffffa1440767>] mq_create+0x337/0x3a0 [dm_cache_mq]
[  665.003139]  [<ffffffffa1436faa>] dm_cache_policy_create+0x4a/0xcc
[dm_cache]
[  665.003143]  [<ffffffffa14339d1>] cache_ctr+0x4a1/0xd40 [dm_cache]
[  665.003152]  [<ffffffffa065dbb8>] ? dm_split_args+0x78/0x140 [dm_mod]
[  665.003159]  [<ffffffffa065ddba>] dm_table_add_target+0x13a/0x390
[dm_mod]
[  665.003166]  [<ffffffffa0660eb0>] table_load+0xd0/0x330 [dm_mod]
[  665.003173]  [<ffffffffa0660de0>] ? table_clear+0xd0/0xd0 [dm_mod]
[  665.003179]  [<ffffffffa0662032>] ctl_ioctl+0x1d2/0x410 [dm_mod]
[  665.003187]  [<ffffffff8103a718>] ? __do_page_fault+0x208/0x4e0
[  665.003193]  [<ffffffffa066227e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
[  665.003198]  [<ffffffff8110bd2e>] do_vfs_ioctl+0x8e/0x4e0
[  665.003202]  [<ffffffff810fbc09>] ? ____fput+0x9/0x10
[  665.003205]  [<ffffffff8110c1d2>] SyS_ioctl+0x52/0x80
[  665.003211]  [<ffffffff815996f9>] system_call_fastpath+0x16/0x1b
[  665.003213] ---[ end trace 2afe5f836777e03f ]---
[  665.008704] device-mapper: table: 253:0: cache: Error creating cache's
policy
[  665.008712] device-mapper: ioctl: error adding target to table


# blockdev --report
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0     26843414528   /dev/sda
rw   256   512  4096       2048      4094689280   /dev/sda1
rw   256   512  4096    7999488     22746759168   /dev/sda2
rw   256   512  4096          0  20973392756736   /dev/sdb
rw   256   512  4096       2048  20973390659584   /dev/sdb1
rw   256   512  4096          0   2397799710720   /dev/sdc
rw   256   512  4096       2048      1023410176   /dev/sdc1
rw   256   512  4096    2000896   2396774203392   /dev/sdc2

regards.



On Fri, Nov 29, 2013 at 3:34 PM, Vladimir Smolensky <arizal@gmail.com>wrote:

> This is complete kernel, right?
>
>
> On Thu, Nov 28, 2013 at 7:28 PM, Joe Thornber <thornber@redhat.com> wrote:
>
>> On Thu, Nov 28, 2013 at 06:50:39PM +0200, Vladimir Smolensky wrote:
>> > Hello, I would like to test the latest dm-cache available to see if the
>> > problem I was having still exists. Where can I get this code?
>>
>> My latest stable code lives in the 'thin-dev' branch of my github tree:
>>
>> https://github.com/jthornber/linux-2.6/tree/thin-dev
>>
>> This has the latest versions of both the thin-provisioning and cache
>> targets, and will shortly have the era target too.
>>
>>
>> - Joe
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>
>

[-- Attachment #1.2: Type: text/html, Size: 6922 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 8197 bytes --]

Hello, the device broke after about a day:

dmesg:

[83086.878182] device-mapper: space map metadata: unable to allocate new
metadata block
[83088.879861] device-mapper: space map metadata: unable to allocate new
metadata block
[83089.414700] device-mapper: space map metadata: unable to allocate new
metadata block
[83089.414706] device-mapper: cache: could not commit metadata for accurate
status
[83090.881533] device-mapper: space map metadata: unable to allocate new
metadata block
[83092.883142] device-mapper: space map metadata: unable to allocate new
metadata block
[83094.884719] device-mapper: space map metadata: unable to allocate new
metadata block
[83096.886355] device-mapper: space map metadata: unable to allocate new
metadata block
[83097.712195] device-mapper: space map metadata: unable to allocate new
metadata block
[83097.712200] device-mapper: cache: could not commit metadata for accurate
status
[83098.283947] device-mapper: space map metadata: unable to allocate new
metadata block
[83098.283952] device-mapper: cache: could not commit metadata for accurate
status
[83098.729396] device-mapper: space map metadata: unable to allocate new
metadata block
[83098.729401] device-mapper: cache: could not commit metadata for accurate
status

dmsetup status:
storage: 0 40963653632 cache 249856/249856 26968230 2678513 711227 30042060
0 248746 248729 0 1 writeback 2 migration_threshold 100000000 4
random_threshold 4 sequential_threshold 10000000

Obviously the metadata partition is full but accourding to the formula I
use it should be more than enough =>  4 MB + ( 16 bytes * nr_blocks ) I get
13341184 ~ 13MB and metadata partiton is 1024MB

device sizes:
 # blockdev --report
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0     26843414528   /dev/sda
rw   256   512  4096       2048      4094689280   /dev/sda1
rw   256   512  4096    7999488     22746759168   /dev/sda2
rw  2048   512  4096          0  20973392756736   /dev/sdb
rw  2048   512  4096       2048  20973390659584   /dev/sdb1
rw  2048   512  4096          0   2397799710720   /dev/sdc
rw  2048   512  4096       2048      1023410176   /dev/sdc1
rw  2048   512  4096    2000896   2396774203392   /dev/sdc2
rw  2048   512   512          0  20973390659584   /dev/dm-0


What am I doing wrong?



regards.



On Fri, Nov 29, 2013 at 5:06 PM, Vladimir Smolensky <arizal@gmail.com>wrote:

> Ok, I compiled it and will test if it works.
>
> When I try to make my cache with blocksize smaller than 8192  I get
>
> # dmsetup  create storage --table '0 40963653632 cache /dev/sdc1 /dev/sdc2
> /dev/sdb1 4096 1 writeback default 0'
> device-mapper: reload ioctl on storage failed: Cannot allocate memory
> Command failed
>
> dnesg:
>
> [  664.940597] device-mapper: cache-policy-mq: version 1.1.0 loaded
> [  665.002924] ------------[ cut here ]------------
> [  665.002943] WARNING: CPU: 0 PID: 18271 at mm/page_alloc.c:2484
> __alloc_pages_nodemask+0x73d/0x820()
> [  665.002945] Modules linked in: dm_cache_mq dm_cache dm_bio_prison
> dm_persistent_data dm_bufio ipv6 bonding coretemp ixgbe i7core_edac
> edac_core kvm processor pcspkr hed button bnx2 dcdbas dca mdio ehci_pci
> thermal_sys microcode joydev sha256_generic libiscsi scsi_transport_iscsi
> tg3 ptp pps_core libphy e1000 fuse nfs lockd sunrpc jfs multipath linear
> raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor xor
> async_tx raid6_pq raid1 raid0 dm_snapshot dm_crypt dm_mirror dm_region_hash
> dm_log dm_mod hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx
> hid_gyration sl811_hcd usbhid ohci_hcd uhci_hcd usb_storage ehci_hcd
> usbcore usb_common mpt2sas raid_class aic94xx libsas lpfc crc_t10dif
> crct10dif_common qla2xxx megaraid_sas megaraid_mbox megaraid_mm megaraid
> aacraid sx8 DAC960 cciss 3w_9xxx 3w_xxxx mptsas scsi_transport_sas mptfc
> scsi_transport_fc scsi_tgt mptspi mptscsih mptbase atp870u dc395x qla1280
> imm parport dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx
> aic79xx scsi_transport_spi sg pdc_adma sata_inic162x sata_mv ata_piix ahci
> libahci sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_nv sata_via
> sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs5530
> pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell
> pata_sc1200 pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd
> pata_ali pata_it8213 pata_ns87415 pata_ns87410 pata_serverworks pata_artop
> pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366
> pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x
> pata_mpiix libata
> [  665.003077] CPU: 0 PID: 18271 Comm: dmsetup Not tainted 3.12.0-rc5+ #1
> [  665.003079] Hardware name: Dell Inc. PowerEdge R510/0DPRKF, BIOS 1.5.3
> 10/25/2010
> [  665.003082]  00000000000009b4 ffff881fada458f8 ffffffff81595198
> 00000000000009b4
> [  665.003085]  0000000000000000 ffff881fada45938 ffffffff81043a62
> ffff881fada45918
> [  665.003088]  0000000000000000 ffff881fae8a0000 0000000000000000
> 0000000000000002
> [  665.003092] Call Trace:
> [  665.003101]  [<ffffffff81595198>] dump_stack+0x49/0x61
> [  665.003107]  [<ffffffff81043a62>] warn_slowpath_common+0x82/0xb0
> [  665.003111]  [<ffffffff81043aa5>] warn_slowpath_null+0x15/0x20
> [  665.003114]  [<ffffffff810c595d>] __alloc_pages_nodemask+0x73d/0x820
> [  665.003120]  [<ffffffffa14405b7>] ? mq_create+0x187/0x3a0 [dm_cache_mq]
> [  665.003124]  [<ffffffff810c5ac2>] __get_free_pages+0x12/0x50
> [  665.003130]  [<ffffffff810f69cb>] __kmalloc+0xeb/0xf0
> [  665.003134]  [<ffffffffa1440767>] mq_create+0x337/0x3a0 [dm_cache_mq]
> [  665.003139]  [<ffffffffa1436faa>] dm_cache_policy_create+0x4a/0xcc
> [dm_cache]
> [  665.003143]  [<ffffffffa14339d1>] cache_ctr+0x4a1/0xd40 [dm_cache]
> [  665.003152]  [<ffffffffa065dbb8>] ? dm_split_args+0x78/0x140 [dm_mod]
> [  665.003159]  [<ffffffffa065ddba>] dm_table_add_target+0x13a/0x390
> [dm_mod]
> [  665.003166]  [<ffffffffa0660eb0>] table_load+0xd0/0x330 [dm_mod]
> [  665.003173]  [<ffffffffa0660de0>] ? table_clear+0xd0/0xd0 [dm_mod]
> [  665.003179]  [<ffffffffa0662032>] ctl_ioctl+0x1d2/0x410 [dm_mod]
> [  665.003187]  [<ffffffff8103a718>] ? __do_page_fault+0x208/0x4e0
> [  665.003193]  [<ffffffffa066227e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
> [  665.003198]  [<ffffffff8110bd2e>] do_vfs_ioctl+0x8e/0x4e0
> [  665.003202]  [<ffffffff810fbc09>] ? ____fput+0x9/0x10
> [  665.003205]  [<ffffffff8110c1d2>] SyS_ioctl+0x52/0x80
> [  665.003211]  [<ffffffff815996f9>] system_call_fastpath+0x16/0x1b
> [  665.003213] ---[ end trace 2afe5f836777e03f ]---
> [  665.008704] device-mapper: table: 253:0: cache: Error creating cache's
> policy
> [  665.008712] device-mapper: ioctl: error adding target to table
>
>
> # blockdev --report
> RO    RA   SSZ   BSZ   StartSec            Size   Device
> rw   256   512  4096          0     26843414528   /dev/sda
> rw   256   512  4096       2048      4094689280   /dev/sda1
> rw   256   512  4096    7999488     22746759168   /dev/sda2
> rw   256   512  4096          0  20973392756736   /dev/sdb
> rw   256   512  4096       2048  20973390659584   /dev/sdb1
> rw   256   512  4096          0   2397799710720   /dev/sdc
> rw   256   512  4096       2048      1023410176   /dev/sdc1
> rw   256   512  4096    2000896   2396774203392   /dev/sdc2
>
> regards.
>
>
>
> On Fri, Nov 29, 2013 at 3:34 PM, Vladimir Smolensky <arizal@gmail.com>wrote:
>
>> This is complete kernel, right?
>>
>>
>> On Thu, Nov 28, 2013 at 7:28 PM, Joe Thornber <thornber@redhat.com>wrote:
>>
>>> On Thu, Nov 28, 2013 at 06:50:39PM +0200, Vladimir Smolensky wrote:
>>> > Hello, I would like to test the latest dm-cache available to see if the
>>> > problem I was having still exists. Where can I get this code?
>>>
>>> My latest stable code lives in the 'thin-dev' branch of my github tree:
>>>
>>> https://github.com/jthornber/linux-2.6/tree/thin-dev
>>>
>>> This has the latest versions of both the thin-provisioning and cache
>>> targets, and will shortly have the era target too.
>>>
>>>
>>> - Joe
>>>
>>> --
>>> dm-devel mailing list
>>> dm-devel@redhat.com
>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>>
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 10619 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: dm-cache fs corruption

[Mike Snitzer]

On Sat, Nov 30 2013 at  9:28am -0500,
Vladimir Smolensky <arizal@gmail.com> wrote:

> Hello, the device broke after about a day:
> 
> dmesg:
> 
> [83086.878182] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83088.879861] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83089.414700] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83089.414706] device-mapper: cache: could not commit metadata for accurate
> status
> [83090.881533] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83092.883142] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83094.884719] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83096.886355] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83097.712195] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83097.712200] device-mapper: cache: could not commit metadata for accurate
> status
> [83098.283947] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83098.283952] device-mapper: cache: could not commit metadata for accurate
> status
> [83098.729396] device-mapper: space map metadata: unable to allocate new
> metadata block
> [83098.729401] device-mapper: cache: could not commit metadata for accurate
> status
> 
> dmsetup status:
> storage: 0 40963653632 cache 249856/249856 26968230 2678513 711227 30042060
> 0 248746 248729 0 1 writeback 2 migration_threshold 100000000 4
> random_threshold 4 sequential_threshold 10000000
> 
> Obviously the metadata partition is full but accourding to the formula I
> use it should be more than enough =>  4 MB + ( 16 bytes * nr_blocks ) I get
> 13341184 ~ 13MB and metadata partiton is 1024MB
>
> device sizes:
>  # blockdev --report
> RO    RA   SSZ   BSZ   StartSec            Size   Device
> rw   256   512  4096          0     26843414528   /dev/sda
> rw   256   512  4096       2048      4094689280   /dev/sda1
> rw   256   512  4096    7999488     22746759168   /dev/sda2
> rw  2048   512  4096          0  20973392756736   /dev/sdb
> rw  2048   512  4096       2048  20973390659584   /dev/sdb1
> rw  2048   512  4096          0   2397799710720   /dev/sdc
> rw  2048   512  4096       2048      1023410176   /dev/sdc1
> rw  2048   512  4096    2000896   2396774203392   /dev/sdc2
> rw  2048   512   512          0  20973390659584   /dev/dm-0
>
> What am I doing wrong?

Not clear yet.  Something is causing you to run out of metadata blocks.

Please provide the output of 'dmsetup table'.  Last time you did the
metadata device that was provided to the cache target was _not_ 1024MB.

Mike

Re: dm-cache fs corruption

[Vladimir Smolensky]

[-- Attachment #1.1: Type: text/plain, Size: 9026 bytes --]

Okay, update on the problem.

Tested with
Linux v-5-231-d1862-150 3.13.0-rc8 #1 SMP Fri Jan 17 17:55:34 GMT 2014
x86_64 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz GenuineIntel GNU/Linux


dm-cache breaks when cache gets full, again

dmesg:

Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589304] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589310] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589312] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589315] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589317] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589319] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589322] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589324] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589326] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589328] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589331] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589333] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589335] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589337] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589339] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589342] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589344] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589346] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589349] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589351] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589353] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589355] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589357] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589360] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589362] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589364] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589366] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589368] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589371] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589373] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589375] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589377] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589448] XFS (dm-0):
metadata I/O error: block 0x2aeb4ee40 ("xfs_trans_read_buf_map") error 117
numblks 16
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589455] XFS (dm-0):
xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590202] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590206] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590209] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590212] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590214] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590216] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590218] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590221] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590223] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590225] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590227] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590229] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590232] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590234] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590236] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590238] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590241] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590243] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590245] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590248] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590250] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590252] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590254] XFS (dm-0):
Corruption detected. Unmount and run xfs_repair

and so on... it just repeats the same

 blockdev --report
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512  4096          0     26843414528   /dev/sda
rw   256   512  4096       2048      4094689280   /dev/sda1
rw   256   512  4096    7999488     22479372288   /dev/sda2
rw  2048   512  4096          0  20973392756736   /dev/sdb
rw  2048   512  4096       2048  20973390659584   /dev/sdb1
rw  2048   512  4096          0   2397799710720   /dev/sdc
rw  2048   512  4096       2048      1023410176   /dev/sdc1
rw  2048   512  4096    2000896   2396774203392   /dev/sdc2
rw  2048   512   512          0  20973390659584   /dev/dm-0


 dmsetup table
storage: 0 40963653632 cache 8:33 8:34 8:17 8192 1 writeback default 0

dmsetup status
storage: 0 40963653632 cache 1152/249856 86805905 9729090 1164931 47563307
0 525209 525209 0 1 writeback 2 migration_threshold 10000000 4
random_threshold 4 sequential_threshold 10000000


Cache size ~2.2T, origin ~20TB

If you need more specific info, please tell  me what.

reagrds.





On Wed, Nov 13, 2013 at 12:24 PM, Vladimir Smolensky <arizal@gmail.com>wrote:

> Hello,
> I've been testing dm-cache for use with static web content.
> It appears that, when using big cache ~3TB(+20TB origin dev) dm-cache
> device corrupts the filesystem after block eviction starts to happen.
> If I set smaller cache size - single 480GB ssd(again with 20TB origin
> dev), the dm-cache dev works just fine.
>
>
> dmesg error:
>
> XFS (dm-0): Corruption detected. Unmount and run xfs_repair
> XFS (dm-0): metadata I/O error: block 0x3d4f534e0
> ("xfs_trans_read_buf_map") error 117 numblks 16
> XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.
> XFS (dm-0): Corruption detected. Unmount and run xfs_repair
> XFS (dm-0): Corruption detected. Unmount and run xfs_repair
>
>
> ext4 also got broken.
>
>
> Tested with 3.10.11-gentoo, fedora 19- kernel 3.11.something, xfs, ext4,
> writeback and writethrough modes.
>
>
>
> regards,
> Vladimir Smolensky
>

[-- Attachment #1.2: Type: text/html, Size: 9996 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]