* btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" @ 2013-06-05 14:57 Martin 2013-06-05 15:05 ` Hugo Mills 0 siblings, 1 reply; 9+ messages in thread From: Martin @ 2013-06-05 14:57 UTC (permalink / raw) To: linux-btrfs Dear Devs, I have x4 4TB HDDs formatted with: mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef] /etc/fstab mounts with the options: noatime,noauto,space_cache,inode_cache All on kernel 3.8.13. Upon using rsync to copy some heavily hardlinked backups from ReiserFS, I've seen: The following "block rsv returned -28" is repeated 7 times until there is a call trace for: "WARNING: at fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad()". Then, the mount is set read-only. How to fix or debug? Thanks, Martin kernel: ------------[ cut here ]------------ kernel: WARNING: at fs/btrfs/extent-tree.c:6372 btrfs_alloc_free_block+0xd3/0x29c() kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: block rsv returned -28 kernel: Modules linked in: raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti Not tainted 3.8.13-gentoo #1 kernel: Call Trace: kernel: [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6 kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel: [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0 kernel: [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c kernel: [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454 kernel: [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56 kernel: [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d kernel: [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624 kernel: [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf kernel: [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6 kernel: [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6 kernel: [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289 kernel: [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5 kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: ---[ end trace b584e8ceb642293f ]--- kernel: ------------[ cut here ]------------ kernel: ------------[ cut here ]------------ kernel: WARNING: at fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad() kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: Transaction aborted kernel: Modules linked in: raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti Tainted: G W 3.8.13-gentoo #1 kernel: Call Trace: kernel: [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2 kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel: [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79 kernel: [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348 kernel: [<ffffffff8142ce4c>] ? commit_fs_roots.isra.25+0xa1/0x14a kernel: [<ffffffff81237a0f>] ? btrfs_scrub_pause+0xd5/0xe4 kernel: [<ffffffff811f4f1a>] ? btrfs_commit_transaction+0x3f9/0x93c kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5 kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: ---[ end trace b584e8ceb6422945 ]--- kernel: BTRFS error (device sdf) in btrfs_save_ino_cache:471: error 28 kernel: btrfs is forced readonly kernel: BTRFS warning (device sdf): Skipping commit of aborted transaction. kernel: BTRFS error (device sdf) in cleanup_transaction:1391: error 28 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 14:57 btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" Martin @ 2013-06-05 15:05 ` Hugo Mills 2013-06-05 15:28 ` Martin 0 siblings, 1 reply; 9+ messages in thread From: Hugo Mills @ 2013-06-05 15:05 UTC (permalink / raw) To: Martin; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 7830 bytes --] On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote: > Dear Devs, > > I have x4 4TB HDDs formatted with: > > mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef] > > > /etc/fstab mounts with the options: > > noatime,noauto,space_cache,inode_cache > > > All on kernel 3.8.13. > > > Upon using rsync to copy some heavily hardlinked backups from ReiserFS, > I've seen: > > > The following "block rsv returned -28" is repeated 7 times until there > is a call trace for: This is ENOSPC. Can you post the output of "btrfs fi df /mountpoint" and "btrfs fi show", please? Hugo. > "WARNING: at fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad()". > > Then, the mount is set read-only. > > > How to fix or debug? > > Thanks, > Martin > > > > kernel: ------------[ cut here ]------------ > kernel: WARNING: at fs/btrfs/extent-tree.c:6372 > btrfs_alloc_free_block+0xd3/0x29c() > kernel: Hardware name: GA-MA790FX-DS5 > kernel: btrfs: block rsv returned -28 > kernel: Modules linked in: raid456 async_raid6_recov async_memcpy > async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow > cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress > sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time > xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat > ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set > ip_set nf_nat > _tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp > nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp > nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp > nf_conntrack_pptp nf_ > conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns > nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 > nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG > nfnetlink_log xt_multiport xt_mar > k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP > xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp > xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 > nf_defrag_i > pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables > x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit > softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea > i2c_algo_bit evdev d > rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr > snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor > button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci > firewire_cor > e pata_atiixp usbhid pata_jmicron sata_sil24 > kernel: Pid: 10980, comm: btrfs-transacti Not tainted 3.8.13-gentoo #1 > kernel: Call Trace: > kernel: [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6 > kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c > kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a > kernel: [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0 > kernel: [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c > kernel: [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454 > kernel: [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56 > kernel: [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d > kernel: [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624 > kernel: [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf > kernel: [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6 > kernel: [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6 > kernel: [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289 > kernel: [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c > kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 > kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 > kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d > kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 > kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5 > kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e > kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 > kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e > kernel: ---[ end trace b584e8ceb642293f ]--- > kernel: ------------[ cut here ]------------ > > > > kernel: ------------[ cut here ]------------ > kernel: WARNING: at fs/btrfs/super.c:256 > __btrfs_abort_transaction+0x3d/0xad() > kernel: Hardware name: GA-MA790FX-DS5 > kernel: btrfs: Transaction aborted > kernel: Modules linked in: raid456 async_raid6_recov async_memcpy > async_pq async_xor xor async_tx raid6_pq act_police cls_basic cls_flow > cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress > sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_LOG xt_time > xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_policy xt_nat > ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set > ip_set nf_nat > _tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp > nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_conntrack_tftp > nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp > nf_conntrack_pptp nf_ > conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns > nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 > nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner xt_NFQUEUE xt_NFLOG > nfnetlink_log xt_multiport xt_mar > k xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP > xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT xt_tcpudp > xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 > nf_defrag_i > pv4 nf_conntrack iptable_mangle nfnetlink iptable_filter ip_tables > x_tables bridge stp llc rtc snd_hda_codec_realtek fbcon bitblit > softcursor font nouveau video mxm_wmi cfbfillrect cfbimgblt cfbcopyarea > i2c_algo_bit evdev d > rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 pcspkr > snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd i2c_core processor > button thermal_sys sky2 wmi backlight fb fbdev pata_acpi firewire_ohci > firewire_cor > e pata_atiixp usbhid pata_jmicron sata_sil24 > kernel: Pid: 10980, comm: btrfs-transacti Tainted: G W > 3.8.13-gentoo #1 > kernel: Call Trace: > kernel: [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2 > kernel: [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c > kernel: [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a > kernel: [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79 > kernel: [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad > kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348 > kernel: [<ffffffff8142ce4c>] ? commit_fs_roots.isra.25+0xa1/0x14a > kernel: [<ffffffff81237a0f>] ? btrfs_scrub_pause+0xd5/0xe4 > kernel: [<ffffffff811f4f1a>] ? btrfs_commit_transaction+0x3f9/0x93c > kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 > kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 > kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d > kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 > kernel: [<ffffffff810420b3>] ? kthread+0xad/0xb5 > kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e > kernel: [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 > kernel: [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e > kernel: ---[ end trace b584e8ceb6422945 ]--- > kernel: BTRFS error (device sdf) in btrfs_save_ino_cache:471: error 28 > kernel: btrfs is forced readonly > kernel: BTRFS warning (device sdf): Skipping commit of aborted transaction. > kernel: BTRFS error (device sdf) in cleanup_transaction:1391: error 28 > > > -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- I'm all for giving people enough rope to shoot themselves in --- the foot -- Andreas Dilger [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 15:05 ` Hugo Mills @ 2013-06-05 15:28 ` Martin 2013-06-05 15:43 ` Hugo Mills 0 siblings, 1 reply; 9+ messages in thread From: Martin @ 2013-06-05 15:28 UTC (permalink / raw) To: linux-btrfs On 05/06/13 16:05, Hugo Mills wrote: > On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote: >> Dear Devs, >> >> I have x4 4TB HDDs formatted with: >> >> mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef] >> >> >> /etc/fstab mounts with the options: >> >> noatime,noauto,space_cache,inode_cache >> >> >> All on kernel 3.8.13. >> >> >> Upon using rsync to copy some heavily hardlinked backups from >> ReiserFS, I've seen: >> >> >> The following "block rsv returned -28" is repeated 7 times until >> there is a call trace for: > > This is ENOSPC. Can you post the output of "btrfs fi df > /mountpoint" and "btrfs fi show", please? btrfs fi df: Data, RAID1: total=2.85TB, used=2.84TB Data: total=8.00MB, used=0.00 System, RAID1: total=8.00MB, used=412.00KB System: total=4.00MB, used=0.00 Metadata, RAID1: total=27.00GB, used=25.82GB Metadata: total=8.00MB, used=0.00 btrfs fi show: Label: 'bu-16TB_0' uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9 Total devices 4 FS bytes used 2.87TB devid 4 size 3.64TB used 1.44TB path /dev/sdf devid 3 size 3.64TB used 1.44TB path /dev/sde devid 1 size 3.64TB used 1.44TB path /dev/sdc devid 2 size 3.64TB used 1.44TB path /dev/sdd And df -h: Filesystem Size Used Avail Use% Mounted on /dev/sde 15T 5.8T 8.9T 40% /mnt/sata16 >> "WARNING: at fs/btrfs/super.c:256 >> __btrfs_abort_transaction+0x3d/0xad()". >> >> Then, the mount is set read-only. >> >> >> How to fix or debug? >> >> Thanks, Martin >> >> >> >> kernel: ------------[ cut here ]------------ kernel: WARNING: at >> fs/btrfs/extent-tree.c:6372 btrfs_alloc_free_block+0xd3/0x29c() >> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: block rsv >> returned -28 kernel: Modules linked in: raid456 async_raid6_recov >> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police >> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb >> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter >> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm >> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG >> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set >> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp >> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 >> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip >> nf_conntrack_proto_udplite nf_conntrack_proto_sctp >> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink >> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc >> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner >> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac >> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP >> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT >> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat >> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle >> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc >> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video >> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d >> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 >> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd >> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev >> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid >> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti >> Not tainted 3.8.13-gentoo #1 kernel: Call Trace: kernel: >> [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6 kernel: >> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel: >> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel: >> [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0 kernel: >> [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c kernel: >> [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454 kernel: >> [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56 kernel: >> [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d kernel: >> [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624 kernel: >> [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf kernel: >> [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6 kernel: >> [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6 kernel: >> [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289 kernel: >> [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c >> kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 >> kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 >> kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d >> kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 kernel: >> [<ffffffff810420b3>] ? kthread+0xad/0xb5 kernel: >> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: >> [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 kernel: >> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: ---[ >> end trace b584e8ceb642293f ]--- kernel: ------------[ cut here >> ]------------ >> >> >> >> kernel: ------------[ cut here ]------------ kernel: WARNING: at >> fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad() >> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: Transaction >> aborted kernel: Modules linked in: raid456 async_raid6_recov >> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police >> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb >> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter >> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm >> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG >> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set >> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp >> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 >> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip >> nf_conntrack_proto_udplite nf_conntrack_proto_sctp >> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink >> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc >> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner >> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac >> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP >> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT >> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat >> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle >> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc >> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video >> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d >> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 >> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd >> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev >> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid >> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti >> Tainted: G W 3.8.13-gentoo #1 kernel: Call Trace: kernel: >> [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2 kernel: >> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel: >> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel: >> [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79 kernel: >> [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad >> kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348 >> kernel: [<ffffffff8142ce4c>] ? >> commit_fs_roots.isra.25+0xa1/0x14a kernel: [<ffffffff81237a0f>] ? >> btrfs_scrub_pause+0xd5/0xe4 kernel: [<ffffffff811f4f1a>] ? >> btrfs_commit_transaction+0x3f9/0x93c kernel: [<ffffffff810427f0>] >> ? abort_exclusive_wait+0x79/0x79 kernel: [<ffffffff811f5a8c>] ? >> start_transaction+0x311/0x408 kernel: [<ffffffff811eed7e>] ? >> transaction_kthread+0xd1/0x16d kernel: [<ffffffff811eecad>] ? >> btrfs_alloc_root+0x34/0x34 kernel: [<ffffffff810420b3>] ? >> kthread+0xad/0xb5 kernel: [<ffffffff81042006>] ? >> __kthread_parkme+0x5e/0x5e kernel: [<ffffffff814315ac>] ? >> ret_from_fork+0x7c/0xb0 kernel: [<ffffffff81042006>] ? >> __kthread_parkme+0x5e/0x5e kernel: ---[ end trace >> b584e8ceb6422945 ]--- kernel: BTRFS error (device sdf) in >> btrfs_save_ino_cache:471: error 28 kernel: btrfs is forced >> readonly kernel: BTRFS warning (device sdf): Skipping commit of >> aborted transaction. kernel: BTRFS error (device sdf) in >> cleanup_transaction:1391: error 28 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 15:28 ` Martin @ 2013-06-05 15:43 ` Hugo Mills 2013-06-05 15:59 ` Martin 2013-06-05 16:24 ` David Sterba 0 siblings, 2 replies; 9+ messages in thread From: Hugo Mills @ 2013-06-05 15:43 UTC (permalink / raw) To: Martin; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 9832 bytes --] On Wed, Jun 05, 2013 at 04:28:33PM +0100, Martin wrote: > On 05/06/13 16:05, Hugo Mills wrote: > > On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote: > >> Dear Devs, > >> > >> I have x4 4TB HDDs formatted with: > >> > >> mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef] > >> > >> > >> /etc/fstab mounts with the options: > >> > >> noatime,noauto,space_cache,inode_cache > >> > >> > >> All on kernel 3.8.13. > >> > >> > >> Upon using rsync to copy some heavily hardlinked backups from > >> ReiserFS, I've seen: > >> > >> > >> The following "block rsv returned -28" is repeated 7 times until > >> there is a call trace for: > > > > This is ENOSPC. Can you post the output of "btrfs fi df > > /mountpoint" and "btrfs fi show", please? > > > btrfs fi df: > > Data, RAID1: total=2.85TB, used=2.84TB > Data: total=8.00MB, used=0.00 > System, RAID1: total=8.00MB, used=412.00KB > System: total=4.00MB, used=0.00 > Metadata, RAID1: total=27.00GB, used=25.82GB > Metadata: total=8.00MB, used=0.00 > > > btrfs fi show: > > Label: 'bu-16TB_0' uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9 > Total devices 4 FS bytes used 2.87TB > devid 4 size 3.64TB used 1.44TB path /dev/sdf > devid 3 size 3.64TB used 1.44TB path /dev/sde > devid 1 size 3.64TB used 1.44TB path /dev/sdc > devid 2 size 3.64TB used 1.44TB path /dev/sdd OK, so you've got plenty of space to allocate. There were some issues in this area (block reserves and ENOSPC, and I think specifically addressing the issue of ENOSPC when there's space available to allocate) that were fixed between 3.8 and 3.9 (and probably some between 3.9 and 3.10-rc as well), so upgrading your kernel _may_ help here. Something else that may possibly help as a sticking-plaster is to write metadata more slowly, so that you don't have quite so much of it waiting to be written out for the next transaction. Practically, this may involve things like running "sync" on a loop. But it's definitely a horrible hack that may help if you're desperate for a quick fix until you can finish creating metadata so quickly and upgrade your kernel... Hugo. > And df -h: > > Filesystem Size Used Avail Use% Mounted on > /dev/sde 15T 5.8T 8.9T 40% /mnt/sata16 > > > > > >> "WARNING: at fs/btrfs/super.c:256 > >> __btrfs_abort_transaction+0x3d/0xad()". > >> > >> Then, the mount is set read-only. > >> > >> > >> How to fix or debug? > >> > >> Thanks, Martin > >> > >> > >> > >> kernel: ------------[ cut here ]------------ kernel: WARNING: at > >> fs/btrfs/extent-tree.c:6372 btrfs_alloc_free_block+0xd3/0x29c() > >> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: block rsv > >> returned -28 kernel: Modules linked in: raid456 async_raid6_recov > >> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police > >> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb > >> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter > >> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm > >> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG > >> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set > >> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp > >> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 > >> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip > >> nf_conntrack_proto_udplite nf_conntrack_proto_sctp > >> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink > >> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc > >> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner > >> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac > >> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP > >> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT > >> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat > >> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle > >> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc > >> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video > >> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d > >> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 > >> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd > >> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev > >> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid > >> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti > >> Not tainted 3.8.13-gentoo #1 kernel: Call Trace: kernel: > >> [<ffffffff811e6600>] ? btrfs_init_new_buffer+0xef/0xf6 kernel: > >> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel: > >> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel: > >> [<ffffffff81278f2c>] ? ___ratelimit+0xc4/0xd0 kernel: > >> [<ffffffff811e66da>] ? btrfs_alloc_free_block+0xd3/0x29c kernel: > >> [<ffffffff811d68e5>] ? __btrfs_cow_block+0x136/0x454 kernel: > >> [<ffffffff811f0d47>] ? btrfs_buffer_uptodate+0x40/0x56 kernel: > >> [<ffffffff811d6d8c>] ? btrfs_cow_block+0x132/0x19d kernel: > >> [<ffffffff811da606>] ? btrfs_search_slot+0x2f5/0x624 kernel: > >> [<ffffffff811dbc5a>] ? btrfs_insert_empty_items+0x5c/0xaf kernel: > >> [<ffffffff811e5089>] ? run_clustered_refs+0x852/0x8e6 kernel: > >> [<ffffffff811e4d20>] ? run_clustered_refs+0x4e9/0x8e6 kernel: > >> [<ffffffff811e7f6b>] ? btrfs_run_delayed_refs+0x10d/0x289 kernel: > >> [<ffffffff811f4ec6>] ? btrfs_commit_transaction+0x3a5/0x93c > >> kernel: [<ffffffff810427f0>] ? abort_exclusive_wait+0x79/0x79 > >> kernel: [<ffffffff811f5a8c>] ? start_transaction+0x311/0x408 > >> kernel: [<ffffffff811eed7e>] ? transaction_kthread+0xd1/0x16d > >> kernel: [<ffffffff811eecad>] ? btrfs_alloc_root+0x34/0x34 kernel: > >> [<ffffffff810420b3>] ? kthread+0xad/0xb5 kernel: > >> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: > >> [<ffffffff814315ac>] ? ret_from_fork+0x7c/0xb0 kernel: > >> [<ffffffff81042006>] ? __kthread_parkme+0x5e/0x5e kernel: ---[ > >> end trace b584e8ceb642293f ]--- kernel: ------------[ cut here > >> ]------------ > >> > >> > >> > >> kernel: ------------[ cut here ]------------ kernel: WARNING: at > >> fs/btrfs/super.c:256 __btrfs_abort_transaction+0x3d/0xad() > >> kernel: Hardware name: GA-MA790FX-DS5 kernel: btrfs: Transaction > >> aborted kernel: Modules linked in: raid456 async_raid6_recov > >> async_memcpy async_pq async_xor xor async_tx raid6_pq act_police > >> cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb > >> sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter > >> xt_statistic xt_CT xt_LOG xt_time xt_connlimit xt_realm > >> xt_addrtype xt_comment xt_recent xt_policy xt_nat ipt_ULOG > >> ipt_REJECT ipt_MASQUERADE ipt_ECN ipt_CLUSTERIP ipt_ah xt_set > >> ip_set nf_nat _tftp nf_nat_snmp_basic nf_conntrack_snmp > >> nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 > >> nf_nat_ftp nf_conntrack_tftp nf_conntrack_sip > >> nf_conntrack_proto_udplite nf_conntrack_proto_sctp > >> nf_conntrack_pptp nf_ conntrack_proto_gre nf_conntrack_netlink > >> nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc > >> nf_conntrack_h323 nf_conntrack_ftp xt_tcpmss xt_pkttype xt_owner > >> xt_NFQUEUE xt_NFLOG nfnetlink_log xt_multiport xt_mar k xt_mac > >> xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP > >> xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_AUDIT > >> xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat > >> nf_conntrack_ipv4 nf_defrag_i pv4 nf_conntrack iptable_mangle > >> nfnetlink iptable_filter ip_tables x_tables bridge stp llc rtc > >> snd_hda_codec_realtek fbcon bitblit softcursor font nouveau video > >> mxm_wmi cfbfillrect cfbimgblt cfbcopyarea i2c_algo_bit evdev d > >> rm_kms_helper snd_hda_intel ttm snd_hda_codec drm i2c_piix4 > >> pcspkr snd_pcm serio_raw snd_page_alloc snd_timer k8temp snd > >> i2c_core processor button thermal_sys sky2 wmi backlight fb fbdev > >> pata_acpi firewire_ohci firewire_cor e pata_atiixp usbhid > >> pata_jmicron sata_sil24 kernel: Pid: 10980, comm: btrfs-transacti > >> Tainted: G W 3.8.13-gentoo #1 kernel: Call Trace: kernel: > >> [<ffffffff811d3600>] ? btrfs_printk+0x12/0xc2 kernel: > >> [<ffffffff810289c8>] ? warn_slowpath_common+0x78/0x8c kernel: > >> [<ffffffff81028a74>] ? warn_slowpath_fmt+0x45/0x4a kernel: > >> [<ffffffff811d5e00>] ? btrfs_release_path+0x5e/0x79 kernel: > >> [<ffffffff811d36ed>] ? __btrfs_abort_transaction+0x3d/0xad > >> kernel: [<ffffffff811ed97b>] ? btrfs_save_ino_cache+0x1d4/0x348 > >> kernel: [<ffffffff8142ce4c>] ? > >> commit_fs_roots.isra.25+0xa1/0x14a kernel: [<ffffffff81237a0f>] ? > >> btrfs_scrub_pause+0xd5/0xe4 kernel: [<ffffffff811f4f1a>] ? > >> btrfs_commit_transaction+0x3f9/0x93c kernel: [<ffffffff810427f0>] > >> ? abort_exclusive_wait+0x79/0x79 kernel: [<ffffffff811f5a8c>] ? > >> start_transaction+0x311/0x408 kernel: [<ffffffff811eed7e>] ? > >> transaction_kthread+0xd1/0x16d kernel: [<ffffffff811eecad>] ? > >> btrfs_alloc_root+0x34/0x34 kernel: [<ffffffff810420b3>] ? > >> kthread+0xad/0xb5 kernel: [<ffffffff81042006>] ? > >> __kthread_parkme+0x5e/0x5e kernel: [<ffffffff814315ac>] ? > >> ret_from_fork+0x7c/0xb0 kernel: [<ffffffff81042006>] ? > >> __kthread_parkme+0x5e/0x5e kernel: ---[ end trace > >> b584e8ceb6422945 ]--- kernel: BTRFS error (device sdf) in > >> btrfs_save_ino_cache:471: error 28 kernel: btrfs is forced > >> readonly kernel: BTRFS warning (device sdf): Skipping commit of > >> aborted transaction. kernel: BTRFS error (device sdf) in > >> cleanup_transaction:1391: error 28 > > > -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- All hope abandon, Ye who press Enter here. --- [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 15:43 ` Hugo Mills @ 2013-06-05 15:59 ` Martin 2013-06-05 16:04 ` Hugo Mills 2013-06-05 16:24 ` David Sterba 1 sibling, 1 reply; 9+ messages in thread From: Martin @ 2013-06-05 15:59 UTC (permalink / raw) To: linux-btrfs On 05/06/13 16:43, Hugo Mills wrote: > On Wed, Jun 05, 2013 at 04:28:33PM +0100, Martin wrote: >> On 05/06/13 16:05, Hugo Mills wrote: >>> On Wed, Jun 05, 2013 at 03:57:42PM +0100, Martin wrote: >>>> Dear Devs, >>>> >>>> I have x4 4TB HDDs formatted with: >>>> >>>> mkfs.btrfs -L bu-16TB_0 -d raid1 -m raid1 /dev/sd[cdef] >>>> >>>> >>>> /etc/fstab mounts with the options: >>>> >>>> noatime,noauto,space_cache,inode_cache >>>> >>>> >>>> All on kernel 3.8.13. >>>> >>>> >>>> Upon using rsync to copy some heavily hardlinked backups >>>> from ReiserFS, I've seen: >>>> >>>> >>>> The following "block rsv returned -28" is repeated 7 times >>>> until there is a call trace for: >>> >>> This is ENOSPC. Can you post the output of "btrfs fi df >>> /mountpoint" and "btrfs fi show", please? >> >> >> btrfs fi df: >> >> Data, RAID1: total=2.85TB, used=2.84TB Data: total=8.00MB, >> used=0.00 System, RAID1: total=8.00MB, used=412.00KB System: >> total=4.00MB, used=0.00 Metadata, RAID1: total=27.00GB, >> used=25.82GB Metadata: total=8.00MB, used=0.00 >> >> >> btrfs fi show: >> >> Label: 'bu-16TB_0' uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9 >> Total devices 4 FS bytes used 2.87TB devid 4 size 3.64TB used >> 1.44TB path /dev/sdf devid 3 size 3.64TB used 1.44TB path >> /dev/sde devid 1 size 3.64TB used 1.44TB path /dev/sdc devid >> 2 size 3.64TB used 1.44TB path /dev/sdd > > OK, so you've got plenty of space to allocate. There were some > issues in this area (block reserves and ENOSPC, and I think > specifically addressing the issue of ENOSPC when there's space > available to allocate) that were fixed between 3.8 and 3.9 (and > probably some between 3.9 and 3.10-rc as well), so upgrading your > kernel _may_ help here. > > Something else that may possibly help as a sticking-plaster is to > write metadata more slowly, so that you don't have quite so much of > it waiting to be written out for the next transaction. Practically, > this may involve things like running "sync" on a loop. But it's > definitely a horrible hack that may help if you're desperate for a > quick fix until you can finish creating metadata so quickly and > upgrade your kernel... > > Hugo. Thanks for that. I can give kernel 3.9.4 a try. For a giggle, I'll try first with "nice 19" and syncs in a loop... One confusing bit is why the "Data, RAID1: total=2.85TB" from "btrfs fi df"? Thanks, Martin ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 15:59 ` Martin @ 2013-06-05 16:04 ` Hugo Mills 0 siblings, 0 replies; 9+ messages in thread From: Hugo Mills @ 2013-06-05 16:04 UTC (permalink / raw) To: Martin; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 1655 bytes --] On Wed, Jun 05, 2013 at 04:59:57PM +0100, Martin wrote: > On 05/06/13 16:43, Hugo Mills wrote: > > On Wed, Jun 05, 2013 at 04:28:33PM +0100, Martin wrote: > >> btrfs fi df: > >> > >> Data, RAID1: total=2.85TB, used=2.84TB Data: total=8.00MB, > >> used=0.00 System, RAID1: total=8.00MB, used=412.00KB System: > >> total=4.00MB, used=0.00 Metadata, RAID1: total=27.00GB, > >> used=25.82GB Metadata: total=8.00MB, used=0.00 > >> > >> > >> btrfs fi show: > >> > >> Label: 'bu-16TB_0' uuid: 8fd9a0a8-9109-46db-8da0-396d9c6bc8e9 > >> Total devices 4 FS bytes used 2.87TB devid 4 size 3.64TB used > >> 1.44TB path /dev/sdf devid 3 size 3.64TB used 1.44TB path > >> /dev/sde devid 1 size 3.64TB used 1.44TB path /dev/sdc devid > >> 2 size 3.64TB used 1.44TB path /dev/sdd > > > > Thanks for that. I can give kernel 3.9.4 a try. For a giggle, I'll try > first with "nice 19" and syncs in a loop... > > > One confusing bit is why the "Data, RAID1: total=2.85TB" from "btrfs > fi df"? Because you've got enough raw space allocated for 2.85 TiB of data; that's 5.7 TiB of actual bytes, because you're using RAID-1 for it. That should add up to somewhere near the total of the "used" values in btrfs fi show. The difference will be accounted for in metadata, system, and the inevitable rounding errors. All the values are shown in powers-of-two -- i.e. IEEE units, not SI units despite the use of SI prefixes. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- All hope abandon, Ye who press Enter here. --- [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 15:43 ` Hugo Mills 2013-06-05 15:59 ` Martin @ 2013-06-05 16:24 ` David Sterba 2013-06-05 21:12 ` Martin 1 sibling, 1 reply; 9+ messages in thread From: David Sterba @ 2013-06-05 16:24 UTC (permalink / raw) To: Hugo Mills, Martin, linux-btrfs On Wed, Jun 05, 2013 at 04:43:29PM +0100, Hugo Mills wrote: > OK, so you've got plenty of space to allocate. There were some > issues in this area (block reserves and ENOSPC, and I think > specifically addressing the issue of ENOSPC when there's space > available to allocate) that were fixed between 3.8 and 3.9 (and > probably some between 3.9 and 3.10-rc as well), so upgrading your > kernel _may_ help here. This is supposed to be fixed by https://patchwork-mail2.kernel.org/patch/2558911/ that went ti 3.10-rc with some followup patches, so it might not be enough as a standalone fix. Unless you really need 'inode_cache', remove it from the mount options. david ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 16:24 ` David Sterba @ 2013-06-05 21:12 ` Martin 2013-06-07 23:14 ` Martin 0 siblings, 1 reply; 9+ messages in thread From: Martin @ 2013-06-05 21:12 UTC (permalink / raw) To: linux-btrfs On 05/06/13 17:24, David Sterba wrote: > On Wed, Jun 05, 2013 at 04:43:29PM +0100, Hugo Mills wrote: >> OK, so you've got plenty of space to allocate. There were some >> issues in this area (block reserves and ENOSPC, and I think >> specifically addressing the issue of ENOSPC when there's space >> available to allocate) that were fixed between 3.8 and 3.9 (and >> probably some between 3.9 and 3.10-rc as well), so upgrading your >> kernel _may_ help here. > > This is supposed to be fixed by > https://patchwork-mail2.kernel.org/patch/2558911/ > > that went ti 3.10-rc with some followup patches, so it might not be > enough as a standalone fix. > > Unless you really need 'inode_cache', remove it from the mount options. Thanks for that. Remounting without the inode_cache option looks to be allowing rsync to continue. (No sync loop needed.) For a 16TB raid1 on kernel 3.8.13, any good mount options to try? For that size of storage and with many hard links, is there any advantage formatting with leaf/node size greater than the default 4kBytes? Thanks, Martin ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" 2013-06-05 21:12 ` Martin @ 2013-06-07 23:14 ` Martin 0 siblings, 0 replies; 9+ messages in thread From: Martin @ 2013-06-07 23:14 UTC (permalink / raw) To: linux-btrfs On 05/06/13 22:12, Martin wrote: > On 05/06/13 17:24, David Sterba wrote: >> On Wed, Jun 05, 2013 at 04:43:29PM +0100, Hugo Mills wrote: >>> OK, so you've got plenty of space to allocate. There were some >>> issues in this area (block reserves and ENOSPC, and I think >>> specifically addressing the issue of ENOSPC when there's space >>> available to allocate) that were fixed between 3.8 and 3.9 (and >>> probably some between 3.9 and 3.10-rc as well), so upgrading your >>> kernel _may_ help here. >> >> This is supposed to be fixed by >> https://patchwork-mail2.kernel.org/patch/2558911/ >> >> that went ti 3.10-rc with some followup patches, so it might not be >> enough as a standalone fix. >> >> Unless you really need 'inode_cache', remove it from the mount options. > > Thanks for that. Remounting without the inode_cache option looks to be > allowing rsync to continue. (No sync loop needed.) rsync is still running ok but the data copying is awfully slow... The copy across is going to take many days at this rate :-( > For a 16TB raid1 on kernel 3.8.13, any good mount options to try? > > For that size of storage and with many hard links, is there any > advantage formatting with leaf/node size greater than the default 4kBytes? Any hints/tips? ;-) Regards, Martin ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-06-07 23:14 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-06-05 14:57 btrfs raid1 on 16TB goes read-only after "btrfs: block rsv returned -28" Martin 2013-06-05 15:05 ` Hugo Mills 2013-06-05 15:28 ` Martin 2013-06-05 15:43 ` Hugo Mills 2013-06-05 15:59 ` Martin 2013-06-05 16:04 ` Hugo Mills 2013-06-05 16:24 ` David Sterba 2013-06-05 21:12 ` Martin 2013-06-07 23:14 ` Martin
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.