All of lore.kernel.org
 help / color / mirror / Atom feed
* btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
@ 2013-06-05 14:57 Martin
  2013-06-05 15:05 ` Hugo Mills
  0 siblings, 1 reply; 9+ messages in thread
From: Martin @ 2013-06-05 14:57 UTC (permalink / raw)
  To: linux-btrfs

Dear Devs,

I have x4 4TB HDDs formatted with:

mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef]


/etc/fstab mounts with the options:

noatime,noauto,space_cache,inode_cache


All on kernel 3.8.13.


Upon using rsync to copy some heavily hardlinked backups from ReiserFS,
I've seen:


The following "block rsv returned -28" is repeated 7 times until there
is a call trace for:

"WARNING: at fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad()".

Then, the mount is set read-only.


How to fix or debug?

Thanks,
Martin



kernel: ------------[ cut here ]------------
kernel: WARNING: at fs/btrfs/extent-tree.c:6372
btrfs_alloc_free_block+0xd3/0x29c()
kernel: Hardware name: GA-MA790FX-DS5
kernel: btrfs: block rsv returned -28
kernel: Modules linked in: raid456 async_raid6_recov async_memcpy
async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow
cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress
sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time
xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat
ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set
ip_set nf_nat
_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp
nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp
nf_conntrack_pptp nf_
conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns
nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323
nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG
nfnetlink_log xt_multiport xt_mar
k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP
xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp
xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
nf_defrag_i
pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables
x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit
softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea
i2c_algo_bit evdev d
rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr
snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor
button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci
firewire_cor
e pata_atiixp usbhid pata_jmicron sata_sil24
kernel: Pid: 10980, comm: btrfs-transacti Not tainted 3.8.13-gentoo #1
kernel: Call Trace:
kernel: [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6
kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c
kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a
kernel: [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0
kernel: [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c
kernel: [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454
kernel: [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56
kernel: [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d
kernel: [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624
kernel: [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf
kernel: [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6
kernel: [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6
kernel: [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289
kernel: [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c
kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79
kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408
kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d
kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34
kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5
kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0
kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
kernel: ---[ end trace b584e8ceb642293f ]---
kernel: ------------[ cut here ]------------



kernel: ------------[ cut here ]------------
kernel: WARNING: at fs/btrfs/super.c:256
__btrfs_abort_transaction+0x3d/0xad()
kernel: Hardware name: GA-MA790FX-DS5
kernel: btrfs: Transaction aborted
kernel: Modules linked in: raid456 async_raid6_recov async_memcpy
async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow
cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress
sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time
xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat
ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set
ip_set nf_nat
_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp
nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp
nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp
nf_conntrack_pptp nf_
conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns
nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323
nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG
nfnetlink_log xt_multiport xt_mar
k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP
xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp
xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
nf_defrag_i
pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables
x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit
softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea
i2c_algo_bit evdev d
rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr
snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor
button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci
firewire_cor
e pata_atiixp usbhid pata_jmicron sata_sil24
kernel: Pid: 10980, comm: btrfs-transacti Tainted: G        W
3.8.13-gentoo #1
kernel: Call Trace:
kernel: [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2
kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c
kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a
kernel: [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79
kernel: [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad
kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348
kernel: [<ffffffff8142ce4c>] ? commit_fs_roots.isra.25+0xa1/0x14a
kernel: [<ffffffff81237a0f>] ? btrfs_scrub_pause+0xd5/0xe4
kernel: [<ffffffff811f4f1a>] ? btrfs_commit_transaction+0x3f9/0x93c
kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79
kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408
kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d
kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34
kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5
kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0
kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
kernel: ---[ end trace b584e8ceb6422945 ]---
kernel: BTRFS error (device sdf) in btrfs_save_ino_cache:471: error 28
kernel: btrfs is forced readonly
kernel: BTRFS warning (device sdf): Skipping commit of aborted transaction.
kernel: BTRFS error (device sdf) in cleanup_transaction:1391: error 28




^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 14:57 btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" Martin
@ 2013-06-05 15:05 ` Hugo Mills
  2013-06-05 15:28   ` Martin
  0 siblings, 1 reply; 9+ messages in thread
From: Hugo Mills @ 2013-06-05 15:05 UTC (permalink / raw)
  To: Martin; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 7830 bytes --]

On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote:
> Dear Devs,
> 
> I have x4 4TB HDDs formatted with:
> 
> mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef]
> 
> 
> /etc/fstab mounts with the options:
> 
> noatime,noauto,space_cache,inode_cache
> 
> 
> All on kernel 3.8.13.
> 
> 
> Upon using rsync to copy some heavily hardlinked backups from ReiserFS,
> I've seen:
> 
> 
> The following "block rsv returned -28" is repeated 7 times until there
> is a call trace for:

   This is ENOSPC. Can you post the output of "btrfs fi df
/mountpoint" and "btrfs fi show", please?

   Hugo.

> "WARNING: at fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad()".
> 
> Then, the mount is set read-only.
> 
> 
> How to fix or debug?
> 
> Thanks,
> Martin
> 
> 
> 
> kernel: ------------[ cut here ]------------
> kernel: WARNING: at fs/btrfs/extent-tree.c:6372
> btrfs_alloc_free_block+0xd3/0x29c()
> kernel: Hardware name: GA-MA790FX-DS5
> kernel: btrfs: block rsv returned -28
> kernel: Modules linked in: raid456 async_raid6_recov async_memcpy
> async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow
> cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress
> sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time
> xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat
> ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set
> ip_set nf_nat
> _tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp
> nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp
> nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp
> nf_conntrack_pptp nf_
> conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns
> nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323
> nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG
> nfnetlink_log xt_multiport xt_mar
> k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP
> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp
> xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
> nf_defrag_i
> pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables
> x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit
> softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea
> i2c_algo_bit evdev d
> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr
> snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor
> button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci
> firewire_cor
> e pata_atiixp usbhid pata_jmicron sata_sil24
> kernel: Pid: 10980, comm: btrfs-transacti Not tainted 3.8.13-gentoo #1
> kernel: Call Trace:
> kernel: [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6
> kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c
> kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a
> kernel: [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0
> kernel: [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c
> kernel: [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454
> kernel: [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56
> kernel: [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d
> kernel: [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624
> kernel: [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf
> kernel: [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6
> kernel: [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6
> kernel: [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289
> kernel: [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c
> kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79
> kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408
> kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d
> kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34
> kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5
> kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
> kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0
> kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
> kernel: ---[ end trace b584e8ceb642293f ]---
> kernel: ------------[ cut here ]------------
> 
> 
> 
> kernel: ------------[ cut here ]------------
> kernel: WARNING: at fs/btrfs/super.c:256
> __btrfs_abort_transaction+0x3d/0xad()
> kernel: Hardware name: GA-MA790FX-DS5
> kernel: btrfs: Transaction aborted
> kernel: Modules linked in: raid456 async_raid6_recov async_memcpy
> async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow
> cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress
> sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time
> xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat
> ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set
> ip_set nf_nat
> _tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp
> nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp
> nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp
> nf_conntrack_pptp nf_
> conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns
> nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323
> nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG
> nfnetlink_log xt_multiport xt_mar
> k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP
> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp
> xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
> nf_defrag_i
> pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables
> x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit
> softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea
> i2c_algo_bit evdev d
> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr
> snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor
> button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci
> firewire_cor
> e pata_atiixp usbhid pata_jmicron sata_sil24
> kernel: Pid: 10980, comm: btrfs-transacti Tainted: G        W
> 3.8.13-gentoo #1
> kernel: Call Trace:
> kernel: [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2
> kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c
> kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a
> kernel: [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79
> kernel: [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad
> kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348
> kernel: [<ffffffff8142ce4c>] ? commit_fs_roots.isra.25+0xa1/0x14a
> kernel: [<ffffffff81237a0f>] ? btrfs_scrub_pause+0xd5/0xe4
> kernel: [<ffffffff811f4f1a>] ? btrfs_commit_transaction+0x3f9/0x93c
> kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79
> kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408
> kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d
> kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34
> kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5
> kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
> kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0
> kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e
> kernel: ---[ end trace b584e8ceb6422945 ]---
> kernel: BTRFS error (device sdf) in btrfs_save_ino_cache:471: error 28
> kernel: btrfs is forced readonly
> kernel: BTRFS warning (device sdf): Skipping commit of aborted transaction.
> kernel: BTRFS error (device sdf) in cleanup_transaction:1391: error 28
> 
> 
> 

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- I'm all for giving people enough rope to shoot themselves in ---   
                       the foot -- Andreas Dilger                        

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 15:05 ` Hugo Mills
@ 2013-06-05 15:28   ` Martin
  2013-06-05 15:43     ` Hugo Mills
  0 siblings, 1 reply; 9+ messages in thread
From: Martin @ 2013-06-05 15:28 UTC (permalink / raw)
  To: linux-btrfs

On 05/06/13 16:05, Hugo Mills wrote:
> On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote:
>> Dear Devs,
>> 
>> I have x4 4TB HDDs formatted with:
>> 
>> mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef]
>> 
>> 
>> /etc/fstab mounts with the options:
>> 
>> noatime,noauto,space_cache,inode_cache
>> 
>> 
>> All on kernel 3.8.13.
>> 
>> 
>> Upon using rsync to copy some heavily hardlinked backups from
>> ReiserFS, I've seen:
>> 
>> 
>> The following "block rsv returned -28" is repeated 7 times until
>> there is a call trace for:
> 
> This is ENOSPC. Can you post the output of "btrfs fi df 
> /mountpoint" and "btrfs fi show", please?


btrfs fi df:

Data, RAID1: total=2.85TB, used=2.84TB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=412.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=27.00GB, used=25.82GB
Metadata: total=8.00MB, used=0.00


btrfs fi show:

Label: 'bu-16TB_0'  uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9
        Total devices 4 FS bytes used 2.87TB
        devid    4 size 3.64TB used 1.44TB path /dev/sdf
        devid    3 size 3.64TB used 1.44TB path /dev/sde
        devid    1 size 3.64TB used 1.44TB path /dev/sdc
        devid    2 size 3.64TB used 1.44TB path /dev/sdd


And df -h:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sde         15T  5.8T  8.9T  40% /mnt/sata16




>> "WARNING: at fs/btrfs/super.c:256
>> __btrfs_abort_transaction+0x3d/0xad()".
>> 
>> Then, the mount is set read-only.
>> 
>> 
>> How to fix or debug?
>> 
>> Thanks, Martin
>> 
>> 
>> 
>> kernel: ------------[ cut here ]------------ kernel: WARNING: at
>> fs/btrfs/extent-tree.c:6372 btrfs_alloc_free_block+0xd3/0x29c() 
>> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: block rsv
>> returned -28 kernel: Modules linked in: raid456 async_raid6_recov
>> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police
>> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb
>> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter
>> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm
>> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG
>> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set 
>> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp
>> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323
>> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip
>> nf_conntrack_proto_udplite nf_conntrack_proto_sctp 
>> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink
>> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc
>> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner
>> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac
>> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP 
>> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT
>> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat
>> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle
>> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc
>> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video
>> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d 
>> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4
>> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd
>> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev
>> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid
>> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti
>> Not tainted 3.8.13-gentoo #1 kernel: Call Trace: kernel:
>> [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6 kernel:
>> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel:
>> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel:
>> [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0 kernel:
>> [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c kernel:
>> [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454 kernel:
>> [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56 kernel:
>> [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d kernel:
>> [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624 kernel:
>> [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf kernel:
>> [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6 kernel:
>> [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6 kernel:
>> [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289 kernel:
>> [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c 
>> kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 
>> kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 
>> kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d 
>> kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 kernel:
>> [<ffffffff810420b3>] ? kthread+0xad/0xb5 kernel:
>> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel:
>> [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 kernel:
>> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: ---[
>> end trace b584e8ceb642293f ]--- kernel: ------------[ cut here
>> ]------------
>> 
>> 
>> 
>> kernel: ------------[ cut here ]------------ kernel: WARNING: at
>> fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad() 
>> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: Transaction
>> aborted kernel: Modules linked in: raid456 async_raid6_recov
>> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police
>> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb
>> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter
>> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm
>> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG
>> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set 
>> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp
>> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323
>> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip
>> nf_conntrack_proto_udplite nf_conntrack_proto_sctp 
>> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink
>> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc
>> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner
>> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac
>> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP 
>> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT
>> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat
>> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle
>> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc
>> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video
>> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d 
>> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4
>> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd
>> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev
>> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid
>> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti
>> Tainted: G        W 3.8.13-gentoo #1 kernel: Call Trace: kernel:
>> [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2 kernel:
>> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel:
>> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel:
>> [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79 kernel:
>> [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad 
>> kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348 
>> kernel: [<ffffffff8142ce4c>] ?
>> commit_fs_roots.isra.25+0xa1/0x14a kernel: [<ffffffff81237a0f>] ?
>> btrfs_scrub_pause+0xd5/0xe4 kernel: [<ffffffff811f4f1a>] ?
>> btrfs_commit_transaction+0x3f9/0x93c kernel: [<ffffffff810427f0>]
>> ? abort_exclusive_wait+0x79/0x79 kernel: [<ffffffff811f5a8c>] ?
>> start_transaction+0x311/0x408 kernel: [<ffffffff811eed7e>] ?
>> transaction_kthread+0xd1/0x16d kernel: [<ffffffff811eecad>] ?
>> btrfs_alloc_root+0x34/0x34 kernel: [<ffffffff810420b3>] ?
>> kthread+0xad/0xb5 kernel: [<ffffffff81042006>] ?
>> __kthread_parkme+0x5e/0x5e kernel: [<ffffffff814315ac>] ?
>> ret_from_fork+0x7c/0xb0 kernel: [<ffffffff81042006>] ?
>> __kthread_parkme+0x5e/0x5e kernel: ---[ end trace
>> b584e8ceb6422945 ]--- kernel: BTRFS error (device sdf) in
>> btrfs_save_ino_cache:471: error 28 kernel: btrfs is forced
>> readonly kernel: BTRFS warning (device sdf): Skipping commit of
>> aborted transaction. kernel: BTRFS error (device sdf) in
>> cleanup_transaction:1391: error 28




^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 15:28   ` Martin
@ 2013-06-05 15:43     ` Hugo Mills
  2013-06-05 15:59       ` Martin
  2013-06-05 16:24       ` David Sterba
  0 siblings, 2 replies; 9+ messages in thread
From: Hugo Mills @ 2013-06-05 15:43 UTC (permalink / raw)
  To: Martin; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 9832 bytes --]

On Wed, Jun 05, 2013 at 04:28:33PM +0100, Martin wrote:
> On 05/06/13 16:05, Hugo Mills wrote:
> > On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote:
> >> Dear Devs,
> >> 
> >> I have x4 4TB HDDs formatted with:
> >> 
> >> mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef]
> >> 
> >> 
> >> /etc/fstab mounts with the options:
> >> 
> >> noatime,noauto,space_cache,inode_cache
> >> 
> >> 
> >> All on kernel 3.8.13.
> >> 
> >> 
> >> Upon using rsync to copy some heavily hardlinked backups from
> >> ReiserFS, I've seen:
> >> 
> >> 
> >> The following "block rsv returned -28" is repeated 7 times until
> >> there is a call trace for:
> > 
> > This is ENOSPC. Can you post the output of "btrfs fi df 
> > /mountpoint" and "btrfs fi show", please?
> 
> 
> btrfs fi df:
> 
> Data, RAID1: total=2.85TB, used=2.84TB
> Data: total=8.00MB, used=0.00
> System, RAID1: total=8.00MB, used=412.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=27.00GB, used=25.82GB
> Metadata: total=8.00MB, used=0.00
> 
> 
> btrfs fi show:
> 
> Label: 'bu-16TB_0'  uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9
>         Total devices 4 FS bytes used 2.87TB
>         devid    4 size 3.64TB used 1.44TB path /dev/sdf
>         devid    3 size 3.64TB used 1.44TB path /dev/sde
>         devid    1 size 3.64TB used 1.44TB path /dev/sdc
>         devid    2 size 3.64TB used 1.44TB path /dev/sdd

   OK, so you've got plenty of space to allocate. There were some
issues in this area (block reserves and ENOSPC, and I think
specifically addressing the issue of ENOSPC when there's space
available to allocate) that were fixed between 3.8 and 3.9 (and
probably some between 3.9 and 3.10-rc as well), so upgrading your
kernel _may_ help here.

   Something else that may possibly help as a sticking-plaster is to
write metadata more slowly, so that you don't have quite so much of it
waiting to be written out for the next transaction. Practically, this
may involve things like running "sync" on a loop. But it's definitely
a horrible hack that may help if you're desperate for a quick fix
until you can finish creating metadata so quickly and upgrade your
kernel...

   Hugo.

> And df -h:
> 
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sde         15T  5.8T  8.9T  40% /mnt/sata16
> 
> 
> 
> 
> >> "WARNING: at fs/btrfs/super.c:256
> >> __btrfs_abort_transaction+0x3d/0xad()".
> >> 
> >> Then, the mount is set read-only.
> >> 
> >> 
> >> How to fix or debug?
> >> 
> >> Thanks, Martin
> >> 
> >> 
> >> 
> >> kernel: ------------[ cut here ]------------ kernel: WARNING: at
> >> fs/btrfs/extent-tree.c:6372 btrfs_alloc_free_block+0xd3/0x29c() 
> >> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: block rsv
> >> returned -28 kernel: Modules linked in: raid456 async_raid6_recov
> >> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police
> >> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb
> >> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter
> >> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm
> >> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG
> >> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set 
> >> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp
> >> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323
> >> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip
> >> nf_conntrack_proto_udplite nf_conntrack_proto_sctp 
> >> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink
> >> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc
> >> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner
> >> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac
> >> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP 
> >> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT
> >> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat
> >> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle
> >> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc
> >> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video
> >> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d 
> >> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4
> >> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd
> >> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev
> >> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid
> >> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti
> >> Not tainted 3.8.13-gentoo #1 kernel: Call Trace: kernel:
> >> [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6 kernel:
> >> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel:
> >> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel:
> >> [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0 kernel:
> >> [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c kernel:
> >> [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454 kernel:
> >> [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56 kernel:
> >> [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d kernel:
> >> [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624 kernel:
> >> [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf kernel:
> >> [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6 kernel:
> >> [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6 kernel:
> >> [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289 kernel:
> >> [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c 
> >> kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 
> >> kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 
> >> kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d 
> >> kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 kernel:
> >> [<ffffffff810420b3>] ? kthread+0xad/0xb5 kernel:
> >> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel:
> >> [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 kernel:
> >> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: ---[
> >> end trace b584e8ceb642293f ]--- kernel: ------------[ cut here
> >> ]------------
> >> 
> >> 
> >> 
> >> kernel: ------------[ cut here ]------------ kernel: WARNING: at
> >> fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad() 
> >> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: Transaction
> >> aborted kernel: Modules linked in: raid456 async_raid6_recov
> >> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police
> >> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb
> >> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter
> >> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm
> >> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG
> >> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set 
> >> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp
> >> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323
> >> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip
> >> nf_conntrack_proto_udplite nf_conntrack_proto_sctp 
> >> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink
> >> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc
> >> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner
> >> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac
> >> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP 
> >> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT
> >> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat
> >> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle
> >> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc
> >> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video
> >> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d 
> >> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4
> >> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd
> >> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev
> >> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid
> >> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti
> >> Tainted: G        W 3.8.13-gentoo #1 kernel: Call Trace: kernel:
> >> [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2 kernel:
> >> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel:
> >> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel:
> >> [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79 kernel:
> >> [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad 
> >> kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348 
> >> kernel: [<ffffffff8142ce4c>] ?
> >> commit_fs_roots.isra.25+0xa1/0x14a kernel: [<ffffffff81237a0f>] ?
> >> btrfs_scrub_pause+0xd5/0xe4 kernel: [<ffffffff811f4f1a>] ?
> >> btrfs_commit_transaction+0x3f9/0x93c kernel: [<ffffffff810427f0>]
> >> ? abort_exclusive_wait+0x79/0x79 kernel: [<ffffffff811f5a8c>] ?
> >> start_transaction+0x311/0x408 kernel: [<ffffffff811eed7e>] ?
> >> transaction_kthread+0xd1/0x16d kernel: [<ffffffff811eecad>] ?
> >> btrfs_alloc_root+0x34/0x34 kernel: [<ffffffff810420b3>] ?
> >> kthread+0xad/0xb5 kernel: [<ffffffff81042006>] ?
> >> __kthread_parkme+0x5e/0x5e kernel: [<ffffffff814315ac>] ?
> >> ret_from_fork+0x7c/0xb0 kernel: [<ffffffff81042006>] ?
> >> __kthread_parkme+0x5e/0x5e kernel: ---[ end trace
> >> b584e8ceb6422945 ]--- kernel: BTRFS error (device sdf) in
> >> btrfs_save_ino_cache:471: error 28 kernel: btrfs is forced
> >> readonly kernel: BTRFS warning (device sdf): Skipping commit of
> >> aborted transaction. kernel: BTRFS error (device sdf) in
> >> cleanup_transaction:1391: error 28
> 
> 
> 

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
           --- All hope abandon,  Ye who press Enter here. ---           

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 15:43     ` Hugo Mills
@ 2013-06-05 15:59       ` Martin
  2013-06-05 16:04         ` Hugo Mills
  2013-06-05 16:24       ` David Sterba
  1 sibling, 1 reply; 9+ messages in thread
From: Martin @ 2013-06-05 15:59 UTC (permalink / raw)
  To: linux-btrfs

On 05/06/13 16:43, Hugo Mills wrote:
> On Wed, Jun 05, 2013 at 04:28:33PM +0100, Martin wrote:
>> On 05/06/13 16:05, Hugo Mills wrote:
>>> On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote:
>>>> Dear Devs,
>>>> 
>>>> I have x4 4TB HDDs formatted with:
>>>> 
>>>> mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef]
>>>> 
>>>> 
>>>> /etc/fstab mounts with the options:
>>>> 
>>>> noatime,noauto,space_cache,inode_cache
>>>> 
>>>> 
>>>> All on kernel 3.8.13.
>>>> 
>>>> 
>>>> Upon using rsync to copy some heavily hardlinked backups
>>>> from ReiserFS, I've seen:
>>>> 
>>>> 
>>>> The following "block rsv returned -28" is repeated 7 times
>>>> until there is a call trace for:
>>> 
>>> This is ENOSPC. Can you post the output of "btrfs fi df 
>>> /mountpoint" and "btrfs fi show", please?
>> 
>> 
>> btrfs fi df:
>> 
>> Data, RAID1: total=2.85TB, used=2.84TB Data: total=8.00MB,
>> used=0.00 System, RAID1: total=8.00MB, used=412.00KB System:
>> total=4.00MB, used=0.00 Metadata, RAID1: total=27.00GB,
>> used=25.82GB Metadata: total=8.00MB, used=0.00
>> 
>> 
>> btrfs fi show:
>> 
>> Label: 'bu-16TB_0'  uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9 
>> Total devices 4 FS bytes used 2.87TB devid    4 size 3.64TB used
>> 1.44TB path /dev/sdf devid    3 size 3.64TB used 1.44TB path
>> /dev/sde devid    1 size 3.64TB used 1.44TB path /dev/sdc devid
>> 2 size 3.64TB used 1.44TB path /dev/sdd
> 
> OK, so you've got plenty of space to allocate. There were some 
> issues in this area (block reserves and ENOSPC, and I think 
> specifically addressing the issue of ENOSPC when there's space 
> available to allocate) that were fixed between 3.8 and 3.9 (and 
> probably some between 3.9 and 3.10-rc as well), so upgrading your 
> kernel _may_ help here.
> 
> Something else that may possibly help as a sticking-plaster is to 
> write metadata more slowly, so that you don't have quite so much of
> it waiting to be written out for the next transaction. Practically,
> this may involve things like running "sync" on a loop. But it's
> definitely a horrible hack that may help if you're desperate for a
> quick fix until you can finish creating metadata so quickly and
> upgrade your kernel...
> 
> Hugo.

Thanks for that. I can give kernel 3.9.4 a try. For a giggle, I'll try
first with "nice 19" and syncs in a loop...


One confusing bit is why the "Data, RAID1: total=2.85TB" from "btrfs
fi df"?

Thanks,
Martin



^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 15:59       ` Martin
@ 2013-06-05 16:04         ` Hugo Mills
  0 siblings, 0 replies; 9+ messages in thread
From: Hugo Mills @ 2013-06-05 16:04 UTC (permalink / raw)
  To: Martin; +Cc: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 1655 bytes --]

On Wed, Jun 05, 2013 at 04:59:57PM +0100, Martin wrote:
> On 05/06/13 16:43, Hugo Mills wrote:
> > On Wed, Jun 05, 2013 at 04:28:33PM +0100, Martin wrote:
> >> btrfs fi df:
> >> 
> >> Data, RAID1: total=2.85TB, used=2.84TB Data: total=8.00MB,
> >> used=0.00 System, RAID1: total=8.00MB, used=412.00KB System:
> >> total=4.00MB, used=0.00 Metadata, RAID1: total=27.00GB,
> >> used=25.82GB Metadata: total=8.00MB, used=0.00
> >> 
> >> 
> >> btrfs fi show:
> >> 
> >> Label: 'bu-16TB_0'  uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9 
> >> Total devices 4 FS bytes used 2.87TB devid    4 size 3.64TB used
> >> 1.44TB path /dev/sdf devid    3 size 3.64TB used 1.44TB path
> >> /dev/sde devid    1 size 3.64TB used 1.44TB path /dev/sdc devid
> >> 2 size 3.64TB used 1.44TB path /dev/sdd
> > 
> 
> Thanks for that. I can give kernel 3.9.4 a try. For a giggle, I'll try
> first with "nice 19" and syncs in a loop...
> 
> 
> One confusing bit is why the "Data, RAID1: total=2.85TB" from "btrfs
> fi df"?

   Because you've got enough raw space allocated for 2.85 TiB of data;
that's 5.7 TiB of actual bytes, because you're using RAID-1 for it.
That should add up to somewhere near the total of the "used" values in
btrfs fi show. The difference will be accounted for in metadata,
system, and the inevitable rounding errors. All the values are shown
in powers-of-two -- i.e. IEEE units, not SI units despite the use of
SI prefixes.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
           --- All hope abandon,  Ye who press Enter here. ---           

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 15:43     ` Hugo Mills
  2013-06-05 15:59       ` Martin
@ 2013-06-05 16:24       ` David Sterba
  2013-06-05 21:12         ` Martin
  1 sibling, 1 reply; 9+ messages in thread
From: David Sterba @ 2013-06-05 16:24 UTC (permalink / raw)
  To: Hugo Mills, Martin, linux-btrfs

On Wed, Jun 05, 2013 at 04:43:29PM +0100, Hugo Mills wrote:
>    OK, so you've got plenty of space to allocate. There were some
> issues in this area (block reserves and ENOSPC, and I think
> specifically addressing the issue of ENOSPC when there's space
> available to allocate) that were fixed between 3.8 and 3.9 (and
> probably some between 3.9 and 3.10-rc as well), so upgrading your
> kernel _may_ help here.

This is supposed to be fixed by
https://patchwork-mail2.kernel.org/patch/2558911/

that went ti 3.10-rc with some followup patches, so it might not be
enough as a standalone fix.

Unless you really need 'inode_cache', remove it from the mount options.


david

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 16:24       ` David Sterba
@ 2013-06-05 21:12         ` Martin
  2013-06-07 23:14           ` Martin
  0 siblings, 1 reply; 9+ messages in thread
From: Martin @ 2013-06-05 21:12 UTC (permalink / raw)
  To: linux-btrfs

On 05/06/13 17:24, David Sterba wrote:
> On Wed, Jun 05, 2013 at 04:43:29PM +0100, Hugo Mills wrote:
>>    OK, so you've got plenty of space to allocate. There were some
>> issues in this area (block reserves and ENOSPC, and I think
>> specifically addressing the issue of ENOSPC when there's space
>> available to allocate) that were fixed between 3.8 and 3.9 (and
>> probably some between 3.9 and 3.10-rc as well), so upgrading your
>> kernel _may_ help here.
> 
> This is supposed to be fixed by
> https://patchwork-mail2.kernel.org/patch/2558911/
> 
> that went ti 3.10-rc with some followup patches, so it might not be
> enough as a standalone fix.
> 
> Unless you really need 'inode_cache', remove it from the mount options.

Thanks for that. Remounting without the inode_cache option looks to be
allowing rsync to continue. (No sync loop needed.)


For a 16TB raid1 on kernel 3.8.13, any good mount options to try?

For that size of storage and with many hard links, is there any
advantage formatting with leaf/node size greater than the default 4kBytes?


Thanks,
Martin


^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28"
  2013-06-05 21:12         ` Martin
@ 2013-06-07 23:14           ` Martin
  0 siblings, 0 replies; 9+ messages in thread
From: Martin @ 2013-06-07 23:14 UTC (permalink / raw)
  To: linux-btrfs

On 05/06/13 22:12, Martin wrote:
> On 05/06/13 17:24, David Sterba wrote:
>> On Wed, Jun 05, 2013 at 04:43:29PM +0100, Hugo Mills wrote:
>>>    OK, so you've got plenty of space to allocate. There were some
>>> issues in this area (block reserves and ENOSPC, and I think
>>> specifically addressing the issue of ENOSPC when there's space
>>> available to allocate) that were fixed between 3.8 and 3.9 (and
>>> probably some between 3.9 and 3.10-rc as well), so upgrading your
>>> kernel _may_ help here.
>>
>> This is supposed to be fixed by
>> https://patchwork-mail2.kernel.org/patch/2558911/
>>
>> that went ti 3.10-rc with some followup patches, so it might not be
>> enough as a standalone fix.
>>
>> Unless you really need 'inode_cache', remove it from the mount options.
> 
> Thanks for that. Remounting without the inode_cache option looks to be
> allowing rsync to continue. (No sync loop needed.)

rsync is still running ok but the data copying is awfully slow... The
copy across is going to take many days at this rate :-(


> For a 16TB raid1 on kernel 3.8.13, any good mount options to try?
> 
> For that size of storage and with many hard links, is there any
> advantage formatting with leaf/node size greater than the default 4kBytes?

Any hints/tips? ;-)


Regards,
Martin




^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-06-07 23:14 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-06-05 14:57 btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" Martin
2013-06-05 15:05 ` Hugo Mills
2013-06-05 15:28   ` Martin
2013-06-05 15:43     ` Hugo Mills
2013-06-05 15:59       ` Martin
2013-06-05 16:04         ` Hugo Mills
2013-06-05 16:24       ` David Sterba
2013-06-05 21:12         ` Martin
2013-06-07 23:14           ` Martin

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.