[linux-lvm] LVM2.2.02.88 - segfault when using lv_t pointers obtained from lvm_vg_list_lvs

[linux-lvm] LVM2.2.02.88 - segfault when using lv_t pointers obtained from lvm_vg_list_lvs

[Alexander Lyakas]

Hi everybody,
we started to work with this LVM version, and sometimes we hit an
issue when using lv_t pointers obtained from lvm_vg_list_lvs() API.
Here are a couple of examples:

#0  0x00007f4487816d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007f4487816d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f448781aa47 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f448784fd7b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f448785cd7a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f448785f31e in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f448784e75f in open_memstream () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007f44878c3deb in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x00007f448893619b in print_log () from /lib/liblvm2app.so.2.2
#8  0x00007f4488948b23 in lv_from_lvid () from /lib/liblvm2app.so.2.2
#9  0x00007f44889139ed in _lv_activate () from /lib/liblvm2app.so.2.2
#10 0x00007f4488914cc4 in lv_activate_with_filter () from /lib/liblvm2app.so.2.2
#11 0x00007f448896718c in _file_lock_resource () from /lib/liblvm2app.so.2.2
#12 0x00007f4488934654 in _lock_vol () from /lib/liblvm2app.so.2.2
#13 0x00007f4488935226 in lock_vol () from /lib/liblvm2app.so.2.2
#14 0x00007f4488910e84 in lvm_lv_activate () from /lib/liblvm2app.so.2.2
...

#0  0x00007fec62e7c1ae in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007fec62e7c1ae in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fec62e7e31e in malloc () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fec62be59de in dm_task_create () from /lib/libdevmapper.so.1.02.1
#3  0x00007fec64430a68 in _setup_task () from /lib/liblvm2app.so.2.2
#4  0x00007fec64430c7a in _info_run () from /lib/liblvm2app.so.2.2
#5  0x00007fec64430e54 in _info () from /lib/liblvm2app.so.2.2
#6  0x00007fec644324de in dev_manager_info () from /lib/liblvm2app.so.2.2
#7  0x00007fec643e5e10 in lv_info () from /lib/liblvm2app.so.2.2
#8  0x00007fec643e782c in lv_deactivate () from /lib/liblvm2app.so.2.2
#9  0x00007fec6443a2ef in _file_lock_resource () from /lib/liblvm2app.so.2.2
#10 0x00007fec64407654 in _lock_vol () from /lib/liblvm2app.so.2.2
#11 0x00007fec64408226 in lock_vol () from /lib/liblvm2app.so.2.2
#12 0x00007fec6440ff27 in lv_remove_single () from /lib/liblvm2app.so.2.2
#13 0x00007fec643e3d13 in lvm_vg_remove_lv () from /lib/liblvm2app.so.2.2
...

Old versions of liblvm we were using did not had the
lvm_lv_from_name() API, so we implemented this using a very
rudimentary code below:
	struct dm_list * lv_list = NULL;
	lv_t lv_handle = NULL;
	lv_list_t * lv_list_entry = NULL;

	lv_list = lvm_vg_list_lvs(vg_handle);
	if (lv_list != NULL) {
		dm_list_iterate_items(lv_list_entry, lv_list) {
			const char * lv_name_from_list = lvm_lv_get_name(lv_list_entry->lv);
			if (strcmp(lv_name_from_list, lv_name) == 0) {
				lv_handle = lv_list_entry->lv;
				break;
			}
		}
	}

This is the code that we obtain the lv_t pointers from. Looking at the
new  lvm_lv_from_name() API, it does exactly the same (without giving
away copies of the list, names etc).  We started using the new API
anyways, and did not see any crashes with it (yet).

Is there anything wrong in the way we obtain lv_t pointers? Could this
be some other issue?

Thanks,
  Alex.

Re: [linux-lvm] LVM2.2.02.88 - segfault when using lv_t pointers obtained from lvm_vg_list_lvs

[Zdenek Kabelac]

2011/9/28 Alexander Lyakas <alex.bolshoy@gmail.com>:
> Hi everybody,
> we started to work with this LVM version, and sometimes we hit an
> issue when using lv_t pointers obtained from lvm_vg_list_lvs() API.
> Here are a couple of examples:
>
> #0 �0x00007f4487816d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) bt
> #0 �0x00007f4487816d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 �0x00007f448781aa47 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 �0x00007f448784fd7b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #3 �0x00007f448785cd7a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #4 �0x00007f448785f31e in malloc () from /lib/x86_64-linux-gnu/libc.so.6
> #5 �0x00007f448784e75f in open_memstream () from /lib/x86_64-linux-gnu/libc.so.6

Looks like you hit some internal glibc malloc  integrity selfcheck -
so some memory has been unexpectedly modified - I'd advise to use
valgrind tool to find proper place when the memory is overwritten.

> � � � �if (lv_list != NULL) {
> � � � � � � � �dm_list_iterate_items(lv_list_entry, lv_list) {
> � � � � � � � � � � � �const char * lv_name_from_list = lvm_lv_get_name(lv_list_entry->lv);
> � � � � � � � � � � � �if (strcmp(lv_name_from_list, lv_name) == 0) {
> � � � � � � � � � � � � � � � �lv_handle = lv_list_entry->lv;
> � � � � � � � � � � � � � � � �break;
> � � � � � � � � � � � �}
> � � � � � � � �}
> � � � �}
>
> This is the code that we obtain the lv_t pointers from. Looking at the
> new �lvm_lv_from_name() API, it does exactly the same (without giving
> away copies of the list, names etc). �We started using the new API
> anyways, and did not see any crashes with it (yet).
>
> Is there anything wrong in the way we obtain lv_t pointers? Could this
> be some other issue?


Depends on, what do you do with pointers - you can't use them once you
free vg handle.
But I think valgrind will be here better then 1000 words....

Zdenek

Re: [linux-lvm] LVM2.2.02.88 - segfault when using lv_t pointers obtained from lvm_vg_list_lvs

[Alexander Lyakas]

Hello Zdenek,

I have done several runs with Valgrind, but unfortunately this problem
did not reproduce yet.

There are, however, several other observations from Valgrind and
non-Valgrind runs.

1) The following warning from Valgrind has appeared a couple of times:
==25539== Syscall param stat(file_name) points to unaddressable byte(s)
==25539==    at 0x61BE715: _xstat (xstat.c:38)
==25539==    by 0x50E016F: _insert (in /lib/liblvm2app.so.2.2)
==25539==    by 0x50E07CD: _full_scan (in /lib/liblvm2app.so.2.2)
==25539==    by 0x50E5DA8: persistent_filter_wipe (in /lib/liblvm2app.so.2.2)
==25539==    by 0x50E11D9: dev_iter_create (in /lib/liblvm2app.so.2.2)
==25539==    by 0x50D765C: lvmcache_label_scan (in /lib/liblvm2app.so.2.2)
==25539==    by 0x5109773: _vg_read (in /lib/liblvm2app.so.2.2)
==25539==    by 0x5109CDB: vg_read_internal (in /lib/liblvm2app.so.2.2)
==25539==    by 0x5109EE4: vg_read (in /lib/liblvm2app.so.2.2)
==25539==    by 0x50D2B74: lvm_vg_open (in /lib/liblvm2app.so.2.2)
==25539==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==25539==

2) Another warning from Valgrind
==25539== Warning: noted but unhandled ioctl 0x1261 with no size/direction hints
==25539==    This could cause spurious value errors to appear.
==25539==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.

3) With the new LVM version (88) there are a lot of error messages like:
semid 32997377: semop failed for cookie 0xd4d4ab7: incorrect semaphore state
Failed to set a proper state for notification semaphore identified by
cookie value 223169207 (0xd4d4ab7) to initialize waiting for incoming
notifications.

These messages mostly turn up after calling lvm_vg_close() after
removing an LV, and sometimes after activating an LV. I did not
compile LVM with debug info yet, to see who causes these messages
there.

4) Internal error: Maps lock 260780032 < unlock 260796416
This message also happens on lvm_vg_close(), right before the previous
error message

5) Additional error that is now frequently seen in syslog
Oct  5 06:49:54 i-0000067c prog: Wiping cache of LVM-capable devices
Oct  5 06:49:54 i-0000067c prog: device-mapper: status ioctl failed:
Invalid argument
Oct  5 06:49:54 i-0000067c prog: Failed to get state of mapped device
Oct  5 06:49:54 i-0000067c prog: device-mapper: status ioctl failed:
Invalid argument
Oct  5 06:49:54 i-0000067c prog: Failed to get state of mapped device
Oct  5 06:49:54 i-0000067c prog: device-mapper: status ioctl failed:
Invalid argument
Oct  5 06:49:54 i-0000067c prog: Failed to get state of mapped device
Oct  5 06:49:54 i-0000067c prog: device-mapper: status ioctl failed:
Invalid argument

I realize that there are several directions to follow here. So I would
really appreciate if you could advise what seems to look like a real
issue out of these. I will also continue trying to reproduce the
original segfault.

Thanks for your help,
  Alex.



On Wed, Sep 28, 2011@2:34 PM, Zdenek Kabelac
<zdenek.kabelac@gmail.com> wrote:
> 2011/9/28 Alexander Lyakas <alex.bolshoy@gmail.com>:
>> Hi everybody,
>> we started to work with this LVM version, and sometimes we hit an
>> issue when using lv_t pointers obtained from lvm_vg_list_lvs() API.
>> Here are a couple of examples:
>>
>> #0 �0x00007f4487816d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb) bt
>> #0 �0x00007f4487816d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1 �0x00007f448781aa47 in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2 �0x00007f448784fd7b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #3 �0x00007f448785cd7a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #4 �0x00007f448785f31e in malloc () from /lib/x86_64-linux-gnu/libc.so.6
>> #5 �0x00007f448784e75f in open_memstream () from /lib/x86_64-linux-gnu/libc.so.6
>
> Looks like you hit some internal glibc malloc �integrity selfcheck -
> so some memory has been unexpectedly modified - I'd advise to use
> valgrind tool to find proper place when the memory is overwritten.
>
>> � � � �if (lv_list != NULL) {
>> � � � � � � � �dm_list_iterate_items(lv_list_entry, lv_list) {
>> � � � � � � � � � � � �const char * lv_name_from_list = lvm_lv_get_name(lv_list_entry->lv);
>> � � � � � � � � � � � �if (strcmp(lv_name_from_list, lv_name) == 0) {
>> � � � � � � � � � � � � � � � �lv_handle = lv_list_entry->lv;
>> � � � � � � � � � � � � � � � �break;
>> � � � � � � � � � � � �}
>> � � � � � � � �}
>> � � � �}
>>
>> This is the code that we obtain the lv_t pointers from. Looking at the
>> new �lvm_lv_from_name() API, it does exactly the same (without giving
>> away copies of the list, names etc). �We started using the new API
>> anyways, and did not see any crashes with it (yet).
>>
>> Is there anything wrong in the way we obtain lv_t pointers? Could this
>> be some other issue?
>
>
> Depends on, what do you do with pointers - you can't use them once you
> free vg handle.
> But I think valgrind will be here better then 1000 words....
>
> Zdenek
>

Re: [linux-lvm] LVM2.2.02.88 - segfault when using lv_t pointers obtained from lvm_vg_list_lvs

[Zdenek Kabelac]

2011/10/9 Alexander Lyakas <alex.bolshoy@gmail.com>:
> Hello Zdenek,
>
> I have done several runs with Valgrind, but unfortunately this problem
> did not reproduce yet.
>
> There are, however, several other observations from Valgrind and
> non-Valgrind runs.
>
> 1) The following warning from Valgrind has appeared a couple of times:
> ==25539== Syscall param stat(file_name) points to unaddressable byte(s)
> ==25539== � �at 0x61BE715: _xstat (xstat.c:38)
> ==25539== � �by 0x50E016F: _insert (in /lib/liblvm2app.so.2.2)

You will surely need to compile lvm  with:
--enable-debug --enable-valgrind-pool




> 2) Another warning from Valgrind
> ==25539== Warning: noted but unhandled ioctl 0x1261 with no size/direction hints
> ==25539== � �This could cause spurious value errors to appear.
> ==25539== � �See README_MISSING_SYSCALL_OR_IOCTL for guidance on
> writing a proper wrapper.

Some ioctl are not known to tje valgrind.

>
> 3) With the new LVM version (88) there are a lot of error messages like:
> semid 32997377: semop failed for cookie 0xd4d4ab7: incorrect semaphore state
> Failed to set a proper state for notification semaphore identified by
> cookie value 223169207 (0xd4d4ab7) to initialize waiting for incoming
> notifications.

Certainly strange


> 4) Internal error: Maps lock 260780032 < unlock 260796416
> This message also happens on lvm_vg_close(), right before the previous
> error message

This maps lock tests are disabled with valgrind compilation, since
valgrind is allocating unexpected memory.


> 5) Additional error that is now frequently seen in syslog
> Oct �5 06:49:54 i-0000067c prog: Wiping cache of LVM-capable devices
> Oct �5 06:49:54 i-0000067c prog: device-mapper: status ioctl failed:
> Invalid argument

Check kernel log (dmesg) what is going on here - possible accessing
non-existing device ?

Zdenek