freeipa

#9312 Nightly test failure (rawhide): add signed zone crashes ods-enforcerd
Closed: fixed by frenaud. Opened by frenaud.

Closed: fixed
Reopen Issue

Issue

The nightly tests running on rawhide detected a new failure related to DNSSEC. See PR #23944 with the following tests:
- test_backup_and_restore_TestBackupAndRestoreWithDNSSEC: Details, report
- test_backup_and_restore_TestBackupReinstallRestoreWithDNSSEC: Details, report
- test_dnssec: Details, report

Steps to Reproduce

  1. install ipa server with DNS server: ipa-server-install --domain ipa.test --realm IPA.TEST -a Secret123 -p Secret123 --setup-dns --forwarder 192.168.122.1 -U
  2. Configure as DNSSEC master: ipa-dns-install --dnssec-master --forwarder 192.168.122.1 -U
  3. Add a signed zone: echo Secret123 | kinit admin ; ipa dnszone-add example.test. --dnssec true

Actual behavior

The zone is not signed and the journal shows that ods-enforcerd crashed:

Jan 29 12:53:48 master.ipa.test systemd-coredump[23412]: Process 23042 (ods-enforcerd) of user 992 dumped core.
                                                         Module libcrypto.so.3 from rpm openssl-3.0.7-4.fc38.x86_64
                                                         Module libssl.so.3 from rpm openssl-3.0.7-4.fc38.x86_64
                                                         Module liblzma.so.5 from rpm xz-5.4.1-1.fc38.x86_64
                                                         Module libz.so.1 from rpm zlib-1.2.13-3.fc38.x86_64
                                                         Module libxml2.so.2 from rpm libxml2-2.10.3-3.fc38.x86_64
                                                         Module libldns.so.3 from rpm ldns-1.8.3-6.fc38.x86_64
                                                         Stack trace of thread 23410:
                                                         #0  0x00007f0540738ad4 __pthread_kill_implementation (libc.so.6 + 0x8ead4)
                                                         #1  0x00007f05406e7aee raise (libc.so.6 + 0x3daee)
                                                         #2  0x00007f05406d087f abort (libc.so.6 + 0x2687f)
                                                         #3  0x00007f05406d160f __libc_message.cold (libc.so.6 + 0x2760f)
                                                         #4  0x00007f05407cca69 __fortify_fail (libc.so.6 + 0x122a69)
                                                         #5  0x00007f05407cb2a4 __chk_fail (libc.so.6 + 0x1212a4)
                                                         #6  0x00007f05407cae85 __snprintf_chk (libc.so.6 + 0x120e85)
                                                         #7  0x0000557876e21462 hsm_generate_rsa_key (ods-enforcerd + 0x4c462)
                                                         #8  0x0000557876deabf7 hsm_key_factory_generate (ods-enforcerd + 0x15bf7)
                                                         #9  0x0000557876dead82 hsm_key_factory_generate_policy (ods-enforcerd + 0x15d82)
                                                         #10 0x0000557876dec336 run.lto_priv.13 (ods-enforcerd + 0x17336)
                                                         #11 0x0000557876e28c7f cmdhandler_handle_client_conversation (ods-enforcerd + 0x53c7f)
                                                         #12 0x0000557876e28e2e cmdhandler_accept_client (ods-enforcerd + 0x53e2e)
                                                         #13 0x0000557876e29017 runthread (ods-enforcerd + 0x54017)
                                                         #14 0x00007f0540736b97 start_thread (libc.so.6 + 0x8cb97)
                                                         #15 0x00007f05407bc9b0 __clone3 (libc.so.6 + 0x1129b0)
                                                         Stack trace of thread 23042:
                                                         #0  0x00007f0540733469 __futex_abstimed_wait_common (libc.so.6 + 0x89469)
                                                         #1  0x00007f0540735e09 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8be09)
                                                         #2  0x0000557876ddfb45 main (ods-enforcerd + 0xab45)
                                                         #3  0x00007f05406d1b4a __libc_start_call_main (libc.so.6 + 0x27b4a)
                                                         #4  0x00007f05406d1c0b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x27c0b)
                                                         #5  0x0000557876de0a75 _start (ods-enforcerd + 0xba75)
                                                         Stack trace of thread 23048:
                                                         #0  0x00007f0540733469 __futex_abstimed_wait_common (libc.so.6 + 0x89469)
                                                         #1  0x00007f0540736172 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8c172)
                                                         #2  0x0000557876e2875c worker_start (ods-enforcerd + 0x5375c)
                                                         #3  0x0000557876e29017 runthread (ods-enforcerd + 0x54017)
                                                         #4  0x00007f0540736b97 start_thread (libc.so.6 + 0x8cb97)
                                                         #5  0x00007f05407bc9b0 __clone3 (libc.so.6 + 0x1129b0)
                                                         Stack trace of thread 23045:
                                                         #0  0x00007f0540733469 __futex_abstimed_wait_common (libc.so.6 + 0x89469)
                                                         #1  0x00007f0540736172 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8c172)
                                                         #2  0x0000557876e2875c worker_start (ods-enforcerd + 0x5375c)
                                                         #3  0x0000557876e29017 runthread (ods-enforcerd + 0x54017)
                                                         #4  0x00007f0540736b97 start_thread (libc.so.6 + 0x8cb97)
                                                         #5  0x00007f05407bc9b0 __clone3 (libc.so.6 + 0x1129b0)
                                                         Stack trace of thread 23047:
                                                         #0  0x00007f0540733469 __futex_abstimed_wait_common (libc.so.6 + 0x89469)
                                                         #1  0x00007f0540736172 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8c172)
                                                         #2  0x0000557876e2875c worker_start (ods-enforcerd + 0x5375c)
                                                         #3  0x0000557876e29017 runthread (ods-enforcerd + 0x54017)
                                                         #4  0x00007f0540736b97 start_thread (libc.so.6 + 0x8cb97)
                                                         #5  0x00007f05407bc9b0 __clone3 (libc.so.6 + 0x1129b0)
                                                         Stack trace of thread 23046:
                                                         #0  0x00007f0540733469 __futex_abstimed_wait_common (libc.so.6 + 0x89469)
                                                         #1  0x00007f0540736172 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8c172)
                                                         #2  0x0000557876e2875c worker_start (ods-enforcerd + 0x5375c)
                                                         #3  0x0000557876e29017 runthread (ods-enforcerd + 0x54017)
                                                         #4  0x00007f0540736b97 start_thread (libc.so.6 + 0x8cb97)
                                                         #5  0x00007f05407bc9b0 __clone3 (libc.so.6 + 0x1129b0)
                                                         Stack trace of thread 23043:
                                                         #0  0x00007f05407b1c87 __select (libc.so.6 + 0x107c87)
                                                         #1  0x0000557876e2aff6 cmdhandler_start (ods-enforcerd + 0x55ff6)
                                                         #2  0x0000557876e29017 runthread (ods-enforcerd + 0x54017)
                                                         #3  0x00007f0540736b97 start_thread (libc.so.6 + 0x8cb97)
                                                         #4  0x00007f05407bc9b0 __clone3 (libc.so.6 + 0x1129b0)
                                                         ELF object binary architecture: AMD x86-64
Jan 29 12:53:48 master.ipa.test systemd[1]: ods-enforcerd.service: Main process exited, code=dumped, status=6/ABRT

Expected behavior

The zone should be signed.

Version/Release/Distribution

Using rawhide:
freeipa-server-4.11.0.dev-0.fc38.x86_64
389-ds-base-2.3.2-1.fc38.x86_64
dogtag-pki-ca-11.2.0-2.fc38.1.noarch
krb5-server-1.20.1-7.fc38.x86_64
opendnssec-2.1.10-4.fc38.x86_64

It is either hsm_hex_unparse() which uses snprintf() or hsm_pkcs11_check_error() with strlcpy() with questionable arguments. We need debuginfo on the host, so a local reproducer would be the best.

Locally reproduced:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007fb56e12ab43 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007fb56e0d9aee in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007fb56e0c287f in __GI_abort () at abort.c:79
#4  0x00007fb56e0c360f in __libc_message (fmt=fmt@entry=0x7fb56e23d2eb "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150
#5  0x00007fb56e1bea69 in __GI___fortify_fail (msg=msg@entry=0x7fb56e23d291 "buffer overflow detected") at fortify_fail.c:24
#6  0x00007fb56e1bd2a4 in __GI___chk_fail () at chk_fail.c:28
#7  0x00007fb56e1bce85 in ___snprintf_chk (s=s@entry=0x7fb5667fa062 "", maxlen=maxlen@entry=33, flag=flag@entry=2, slen=slen@entry=31, format=format@entry=0x55bca8052ed5 "%02x") at snprintf_chk.c:29
#8  0x000055bca8041462 in snprintf (__fmt=0x55bca8052ed5 "%02x", __n=33, __s=0x7fb5667fa062 "") at /usr/include/bits/stdio2.h:54
#9  hsm_hex_unparse (len=16, src=0x7fb5667fa270 "\332\004\216mE\312\037\026(L\272\310@\023\376Q\002", dst=0x7fb5667fa060 "da") at ../../libhsm/src/lib/libhsm.c:1385
#10 hsm_generate_rsa_key (ctx=0x7fb54c02fd80, repository=<optimized out>, keysize=<optimized out>) at ../../libhsm/src/lib/libhsm.c:2674
#11 0x000055bca800abf7 in hsm_key_factory_generate (engine=engine@entry=0x55bca9bbc820, connection=connection@entry=0x7fb54c075e80, policy=policy@entry=0x7fb54c005e00, policy_key=0x7fb54c00a430, duration=<optimized out>, 
    duration@entry=0) at hsmkey/hsm_key_factory.c:256
#12 0x000055bca800ad82 in hsm_key_factory_generate_policy (engine=engine@entry=0x55bca9bbc820, connection=connection@entry=0x7fb54c075e80, policy=policy@entry=0x7fb54c005e00, duration=duration@entry=0) at hsmkey/hsm_key_factory.c:379
#13 0x000055bca800c336 in run (sockfd=<optimized out>, context=<optimized out>, cmd=<optimized out>) at keystate/zone_add_cmd.c:399
#14 0x000055bca8048c7f in cmdhandler_perform_command (context=0x7fb568004440, cmd=0x7fb5667fb540 "zone add --zone example.test --input /var/lib/ipa/dns/zone/entryUUID/faa42581-a076-11ed-bf06-f5a75fa133ef")
    at ../../common/cmdhandler.c:129
#15 extract_msg (buflen=1024, context=<optimized out>, exitcode=<synthetic pointer>, pos=<synthetic pointer>, buf=0x7fb5667fb950 "\002") at ../../common/cmdhandler.c:198
#16 cmdhandler_handle_client_conversation (context=context@entry=0x7fb568004440) at ../../common/cmdhandler.c:250
#17 0x000055bca8048e2e in cmdhandler_accept_client (arg=0x7fb568004440) at ../../common/cmdhandler.c:284
#18 0x000055bca8049017 in runthread (data=0x7fb568004470) at ../../common/janitor.c:318
#19 0x00007fb56e128b97 in start_thread (arg=<optimized out>) at pthread_create.c:444
#20 0x00007fb56e1ae9b0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

I saved the core file if you want to have a look.

Thank you. So this is hsm_hex_unparse() as I thought. In the 

#7  0x00007fb56e1bce85 in ___snprintf_chk (s=s@entry=0x7fb5667fa062 "", maxlen=maxlen@entry=33, flag=flag@entry=2, slen=slen@entry=31, format=format@entry=0x55bca8052ed5 "%02x") at snprintf_chk.c:29

In https://github.com/bminor/glibc/blob/master/debug/snprintf_chk.c we get

  if (__glibc_unlikely (slen < maxlen))
    __chk_fail ();

When slen == 31 and maxlen == 33, we trigger __chk_fail(), hence a crash.

The slen comes from bits/stdio2.h:

# ifdef __va_arg_pack
__fortify_function int
__NTH (snprintf (char *__restrict __s, size_t __n,
                 const char *__restrict __fmt, ...))
{
  return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
                                   __glibc_objsize (__s), __fmt,
                                   __va_arg_pack ());
}
# elif !defined __cplusplus
#  define snprintf(str, len, ...) \
  __builtin___snprintf_chk (str, len, __USE_FORTIFY_LEVEL - 1,                \
                            __glibc_objsize (str), __VA_ARGS__)
# endif

slen is the string object size as seen by glibc. In case of FORTIFY level 3 and newer GCC, it is using __builtin_dynamic_object_size, otherwise it would use __builtin_object_size. GCC says about this: https://gcc.gnu.org/onlinedocs/gcc/Object-Size-Checking.html

Judging by the mass rebuild logs, it looks like when source fortification is level 3, we are consistently losing at least couple bytes in those estimations, so we get warnings like these ones:

make[3]: Entering directory '/builddir/build/BUILD/opendnssec-2.1.10/libhsm/src/lib'
gcc -DHAVE_CONFIG_H -I. -I../../../common  -I../../../common -I../../../common -I./cryptoki_compat -I/usr/lib64/include -I/usr/include/libxml2  -std=c99 -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fPIE -pie -Wextra -Wformat -Wformat-nonliteral -Wformat-security -pedantic -c -o libhsm.o libhsm.c
make[3]: Leaving directory '/builddir/build/BUILD/opendnssec-2.1.10/libhsm/src/lib'
libhsm.c: In function 'hsm_pkcs11_load_functions':
libhsm.c:293:28: warning: ISO C forbids conversion of object pointer to function pointer type [-Wpedantic]
  293 |         pGetFunctionList = (CK_C_GetFunctionList) dlsym(pDynLib, "C_GetFunctionList");
      |                            ^
libhsm.c: In function 'hsm_get_key_info':
libhsm.c:3230:18: warning: '%lu' directive output may be truncated writing between 1 and 20 bytes into a region of size 16 [-Wformat-truncation=]
 3230 |                 "%lu", key_info->algorithm);
      |                  ^~~
libhsm.c:3230:17: note: directive argument in the range [2, 18446744073709551615]
 3230 |                 "%lu", key_info->algorithm);
      |                 ^~~~~
In file included from /usr/include/stdio.h:906,
                 from libhsm.c:30:
In function 'snprintf',
    inlined from 'hsm_get_key_info' at libhsm.c:3229:13:
/usr/include/bits/stdio2.h:54:10: note: '__snprintf_chk' output between 2 and 21 bytes into a destination of size 16
   54 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   55 |                                    __glibc_objsize (__s), __fmt,
      |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   56 |                                    __va_arg_pack ());
      |                                    ~~~~~~~~~~~~~~~~~
libhsm.c: In function 'hsm_get_error':
libhsm.c:3533:18: warning: '%s' directive output may be truncated writing up to 511 bytes into a region of size between 501 and 510 [-Wformat-truncation=]
 3533 |             "%s: %s",
      |                  ^~
In function 'snprintf',
    inlined from 'hsm_get_error' at libhsm.c:3532:9:
/usr/include/bits/stdio2.h:54:10: note: '__snprintf_chk' output 3 or more bytes (assuming 523) into a destination of size 512
   54 |   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   55 |                                    __glibc_objsize (__s), __fmt,
      |                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   56 |                                    __va_arg_pack ());
      |                                    ~~~~~~~~~~~~~~~~~
make[3]: Entering directory '/builddir/build/BUILD/opendnssec-2.1.10/libhsm/src/lib'

An easiest solution would be to bump id_str size. In OpenDNSSEC's code, it is defined in hsm_generate_rsa_key() as

    /* ids we create are 16 bytes of data */
    unsigned char id[16];
    /* that's 33 bytes in string (16*2 + 1 for \0) */
    char id_str[33];

Actually, we can simply pass the right buffer size in hsm_hex_unparse() to snprintf(). I submitted https://github.com/opendnssec/opendnssec/pull/842 upstream.

Should be fixed with https://koji.fedoraproject.org/koji/taskinfo?taskID=96872362
@frenaud could you please try it?

@abbra
I installed the opendnssec build from the above link, zone signing is now working:
dig @IP_ADDR -t dnskey example.test. now returns the dnskey records.

Closing as fixed, the last run with rawhide and opendnssec-2.1.10-5.fc38.x86_64 was successful (see PR #2430).
Edited by frenaud

Metadata Update from @frenaud:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
Metadata
None

test-failure tracker

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.