#63 Segfault in thr_stack_create()
Closed: Fixed Opened by firstyear.

{{{
(gdb) run
Starting program: /home/william/build/nunc-stans/.libs/lt-cmocka_stress_test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[==========] Running 1 test(s).
[ RUN ] ns_stress_test
7 ns_thrpool_new(): max threads, (80)
stacksize (0), event q size (unbounded), work q size (unbounded)

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff79c83aa in lfds710_stack_init_valid_on_current_logical_core (ss=0x6054e0, user_state=0x0) at ../../src/lfds710_stack/lfds710_stack_init.c:13
13 LFDS710_PAL_ASSERT( (lfds710_pal_uint_t) ss->top % LFDS710_PAL_ATOMIC_ISOLATION_IN_BYTES == 0 );
Missing separate debuginfos, use: dnf debuginfo-install libcmocka-1.1.0-3.fc26.x86_64 nss-softokn-freebl-3.26.0-4.fc26.x86_64
(gdb) bt

0 0x00007ffff79c83aa in lfds710_stack_init_valid_on_current_logical_core (ss=0x6054e0, user_state=0x0) at ../../src/lfds710_stack/lfds710_stack_init.c:13

1 0x00007ffff79c381c in thr_stack_create () at /home/william/development/389ds/nunc-stans/ns_thrpool.c:234

2 0x00007ffff79c6096 in ns_thrpool_new (tp_config=0x7fffffffe320) at /home/william/development/389ds/nunc-stans/ns_thrpool.c:1504

3 0x00000000004023e5 in ns_stress_test (state=0x605020) at /home/william/development/389ds/nunc-stans/tests/cmocka/stress_test.c:461

4 0x00007ffff7bd1c98 in cmocka_run_one_test_or_fixture () from /lib64/libcmocka.so.0

5 0x00007ffff7bd25b7 in _cmocka_run_group_tests () from /lib64/libcmocka.so.0

6 0x00000000004027f8 in main () at /home/william/development/389ds/nunc-stans/tests/cmocka/stress_test.c:554

(gdb)
}}}

Without asan, segfault occurs on stress test.

However, the struct appears correct:

{{{
(gdb) print *ss
$2 = {top = {0x0, 0x0}, user_state = 0x0, pop_backoff = {lock = 0, backoff_iteration_frequency_counters = {0, 0}, metric = 0, total_operations = 0}, push_backoff = {lock = 0,
backoff_iteration_frequency_counters = {0, 0}, metric = 0, total_operations = 0}}
(gdb) print ss->top
$3 = {0x0, 0x0}
}}}


So the fault is from

{{{

LFDS710_PAL_ASSERT( (lfds710_pal_uint_t) ss->top % LFDS710_PAL_ATOMIC_ISOLATION_IN_BYTES == 0 );

}}}

Where the ASSERT fails. So ss->top == 0, which means that ss>top % 0x80 == 0. So this is not correct, something weird here ...

{{{

include

int
main(int argc, char **argv) {
printf("%d\n", !(0 % 128 == 0));
return 0;
}

}}}

Which correctly prints 0, and the assert macro is:

{{{

define LFDS710_PAL_ASSERT( expression ) if( !(expression) ) LFDS710_MISC_DELIBERATELY_CRASH;

}}}

So give that if will execute the code if nonzero, something funky is happening here.

Ahhhh I see. I was mislead by gdb. {0x0, 0x0} is because it knows the array is of fixed size. So what's actually happening is that ss->top % LFDS710_PAL_ATOMIC_ISOLATION_IN_BYTES is checking the address of the array, but gdb "helpfully" is just showing the content.

This yields

{{{
(gdb) print &ss->top
$3 = (struct lfds710_stack_element * volatile(*)[2]) 0x6054e0
(gdb) print 0x6054e0 % 128
$4 = 96
}}}

This then triggers the assert, because the library won't be able to CAS on an address that isn't aligned a 128byte boundary.

Curiously, this doesn't affect a build that uses ASAN ...

This appears to be an issue between Fedora gcc vs Rhel GCC

Affects - Fedora: gcc-6.2.1-2.fc26.x86_64

Does not affect - Rhel: gcc-4.8.5-4.el7.x86_64

Okay, its not an issue in GCC. It's a flaw in my understanding of the alignment operator. This only affects stack allocated elements. It was by chance that previously on el7 this would allocate on a 128 byte alignment. It just so happens that gcc 6 produces different code, and thus the issue.

So as we are mallocing on the heap, the stack state and queue states for lfds, these must be aligned to a 128 byte boundary, but malloc doesn't guarantee this! How does malloc know the struct wants alignment at runtime. It doesn't.

So we need to use the posix_memalign functions to satisfy this.

This means we'll need to add a similar memalign function to slapi_ch_ to satisfy this.

commit a11fb3a4848d0a04753abd27b651fa9d0e0bfd15
Compressing objects: 100% (33/33), done.
Writing objects: 100% (33/33), 9.68 KiB | 0 bytes/s, done.
Total 33 (delta 25), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/nunc-stans.git
9dacc22..59aa673 master -> master

Metadata