freeipa

#9456 ipa-replica-install RuntimeError: Failed to start replication on RHEL9
Closed: fixed by zlopez. Opened by zlopez.

Closed: fixed
Reopen Issue

Issue

I'm trying to update Fedora staging IPA from RHEL8 to RHEL9 following this guide and when running ipa-replica-install I'm getting the RuntimeError: Failed to start replication.

I'm doing all this in Fedora staging environment and I'm using our playbook to deploy the machine. I'm running only the make the box real task without ipa/client role.

Steps to Reproduce

  1. Run the ansible playbook to deploy RHEL 9 VM
  2. Run 
ipa-replica-install --setup-ca --setup-kra --admin-password=XXX --no-host-dns --mkhomedir --no-ntp --unattended --no-ssh --no-sshd --force-join --log-file=/var/log/ipainstall.log --domain=STG.FEDORAPROJECT.ORG --server=ipa01.stg.iad2.fedoraproject.org

Actual behavior

[error] RuntimeError: Failed to start replication

Expected behavior

Replica created

Version/Release/Distribution

$ rpm -q freeipa-server freeipa-client ipa-server ipa-client 389-ds-base pki-ca krb5-server

package freeipa-server is not installed
package freeipa-client is not installed
ipa-server-4.10.1-9.el9_2.x86_64
ipa-client-4.10.1-9.el9_2.x86_64
389-ds-base-2.2.4-5.el9_2.x86_64
package pki-ca is not installed
krb5-server-1.20.1-9.el9_2.x86_64

Additional info:

Here is the output from ipa-replica-install command:

Configuring client side components
This program will set up IPA client.           
Version 4.10.1                                                                                                                                                                                
Client hostname: ipa03.stg.iad2.fedoraproject.org                                              
Realm: STG.FEDORAPROJECT.ORG                                                                   
DNS Domain: stg.fedoraproject.org                                                              
IPA Server: ipa01.stg.iad2.fedoraproject.org                                                                                                                                                  
BaseDN: dc=stg,dc=fedoraproject,dc=org 
Skipping chrony configuration   
Successfully retrieved CA cert                                                                 
    Subject:     CN=Certificate Authority,O=STG.FEDORAPROJECT.ORG
    Issuer:      CN=Certificate Authority,O=STG.FEDORAPROJECT.ORG
    Valid From:  2021-02-11 13:11:00
    Valid Until: 2041-02-11 13:11:00
Enrolled in IPA realm STG.FEDORAPROJECT.ORG                                                                                                                                                   
Created /etc/ipa/default.conf                                                                                                                                                                 
Configured /etc/sssd/sssd.conf                                                                                                                                                                
Systemwide CA database updated.                                                                                                                                                               
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub                                                                                                                                    
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub                                                                                                                                  
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub                                                                                                                                      
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key-cert.pub                                                                                                                               
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key-cert.pub                                                                                                                             
Adding SSH public key from /etc/ssh/ssh_host_rsa_key-cert.pub                                                                                                                                 
Could not update DNS SSHFP records.                                                                                                                                                           
SSSD enabled                                                                                                                                                                                  
Configured /etc/openldap/ldap.conf                                                                                                                                                            
Configuring stg.fedoraproject.org as NIS domain.                                                                                                                                              
Configured /etc/krb5.conf for IPA realm STG.FEDORAPROJECT.ORG                                                                                                                                 
Client configuration complete.                                                                                                                                                                
The ipa-client-install command was successful                                                                                                                                                 
Warning: skipping DNS resolution of host ipa03.stg.iad2.fedoraproject.org                                                                                                                     
Warning: skipping DNS resolution of host ipa01.stg.iad2.fedoraproject.org                                                                                                                     
Lookup failed: Preferred host ipa03.stg.iad2.fedoraproject.org does not provide DNS.                                                                                                          
Trust is configured but no NetBIOS domain name found, setting it now.                                                                                                                         
WARNING: 73612 existing users or groups do not have a SID identifier assigned.                                                                                                                
Installer can run a task to have ipa-sidgen Directory Server plugin generate                   
the SID identifier for all these users. Please note, in case of a high                         
number of users and groups, the operation might lead to high replication                                                                                                                      
traffic and performance degradation. Refer to ipa-adtrust-install(1) man page                                                                                                                 
for details.                      
Unattended mode was selected, installer will NOT run ipa-sidgen task!                                                                                                                         
Run connection check to master                                                                 
Connection check OK                                                                            
Disabled p11-kit-proxy                                                                         
Configuring directory server (dirsrv). Estimated time: 30 seconds                              
  [1/39]: creating directory server instance                                                                                                                                                  
Validate installation settings ...     
Create file system structures ...              
Perform SELinux labeling ...    
Create database backend: dc=stg,dc=fedoraproject,dc=org ...                                    
Perform post-installation tasks ...                                                            
  [2/39]: tune ldbm plugin                                                                     
  [3/39]: adding default schema     
  [4/39]: enabling memberof plugin  
  [5/39]: enabling winsync plugin                                                                                                                                                             
  [6/39]: configure password logging                                                                                                                                                          
  [7/39]: configuring replication version plugin                                                                                                                                              
  [8/39]: enabling IPA enrollment plugin                                                                                                                                                      
  [9/39]: configuring uniqueness plugin                                                                                                                                                       
  [10/39]: configuring uuid plugin                                                                                                                                                            
  [11/39]: configuring modrdn plugin                                                                                                                                                          
  [12/39]: configuring DNS plugin                                                                                                                                                             
  [13/39]: enabling entryUSN plugin                                                                                                                                                           
  [14/39]: configuring lockout plugin                                                                                                                                                         
  [15/39]: configuring graceperiod plugin                                                                                                                                                     
  [16/39]: configuring topology plugin                                                                                                                                                        
  [17/39]: creating indices                                                                                                                                                                   
  [18/39]: enabling referential integrity plugin                                                                                                                                              
  [19/39]: configuring certmap.conf                                                                                                                                                           
  [20/39]: configure new location for managed entries                                                                                                                                         
  [21/39]: configure dirsrv ccache and keytab                                                                                                                                                 
  [22/39]: enabling SASL mapping fallback                                                                                                                                                     
  [23/39]: restarting directory server                                                                                                                                                        
  [24/39]: creating DS keytab                                                                                                                                                                 
  [25/39]: ignore time skew for initial replication                                                                                                                                           
  [26/39]: setting up initial replication                                                                                                                                                     
Starting replication, please wait until this has completed.                                                                                                                                   
Update in progress, 14 seconds elapsed                                                                                                                                                        
[ldap://ipa01.stg.iad2.fedoraproject.org:389] reports: Update failed! Status: [Error (-11) connection error: Unknown connection error (-11) - Total update aborted]                           
  [error] RuntimeError: Failed to start replication                                            
Your system may be partly configured.                                                                                                                                                         
Run /usr/sbin/ipa-server-install --uninstall to clean up.                                                                                                                                     
Failed to start replication                    
The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

And here is the log file.

The issue could be somewhere in Fedora staging deployment, but I'm not sure where exactly. Error messages I'm getting are not saying much.

Log file locations: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/config-files-logs.html
Troubleshooting guide: https://www.freeipa.org/page/Troubleshooting

Found something in ipa01 log:

Sep 27 14:21:23 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: [27/Sep/2023:14:21:23.518893204 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa03.stg.iad2.fedoraproject.org" (ipa03:389) - Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) ()
Sep 27 14:21:26 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/STG.IAD2.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG not found in Kerberos database)
Sep 27 14:21:32 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: [27/Sep/2023:14:21:32.618342270 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa03.stg.iad2.fedoraproject.org" (ipa03:389) - Replication bind 
with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
Sep 27 13:42:02 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: [27/Sep/2023:13:42:02.979456798 +0000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToipa03.stg.iad2.fedoraproject.org" (ipa03:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.

EDIT: Adding more related lines.
Edited by zlopez

Found something in ipa01 log:
Sep 27 14:21:23 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: [27/Sep/2023:14:21:23.518893204 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa03.stg.iad2.fedoraproject.org" (ipa03:389) - Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) () Sep 27 14:21:26 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server krbtgt/STG.IAD2.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG not found in Kerberos database) Sep 27 14:21:32 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: [27/Sep/2023:14:21:32.618342270 +0000] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa03.stg.iad2.fedoraproject.org" (ipa03:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) () Sep 27 13:42:02 ipa01.stg.iad2.fedoraproject.org ns-slapd[2437722]: [27/Sep/2023:13:42:02.979456798 +0000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToipa03.stg.iad2.fedoraproject.org" (ipa03:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica.

EDIT: Adding more related lines.

Hi @zlopez
the error The remote replica has a different database generation ID than the local database can be ignored as the replication hasn't started yet.

The other error Server krbtgt/STG.IAD2.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG not found in Kerberos database) is not a familiar one. I would expect the principal to be krbtgt/STG.FEDORAPROJECT.ORG@STG.FEDORAPROJECT.ORG instead but the logs show an IAD2 component. Does the server ipa01 have multiple aliases? Was it installed with realm=STG.FEDORAPROJECT.ORG and domain.stg.fedoraproject.org?

Looking at the configuration of ipa01 and it has domain and realm set to stg.fedoraproject.org, no iad2 anywhere. And the same is set in case of ipa03.

I'm not sure where this is coming from. How can I check if there are any aliases?

Any advice how to move this forward?

We found the issue behind this. See https://pagure.io/fedora-infrastructure/issue/10358#comment-880274 for more info.
The issue was with sasl packet exceeding the max size.

I'm closing this one as the issue wasn't related to IPA itself

Metadata Update from @zlopez:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)
Metadata
None
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.