Directory Server QE : Guidelines for using pytest and lib389

The guide covers basic workflow with git, py.test and 1minutetip CLIs; guidance with python-pytest module, lib389 and python-ldap features.

For a saving place purposes, I'll replace topology_m2.ms["master1"] with master1, etc.

Basic workflow

  1. Clone ds repo:

    1. git clone ssh://git@pagure.io/389-ds-base.git

    2. As an option you can create your repo and work with it before pushing to upstream: 

      • https://mojo.redhat.com/docs/DOC-982603

        ssh shell.devel.redhat.com
        mkdir -p ~/public_git 
        chmod755 ~/public_git
        cd ~/public_git
        git clone --bare https://pagure.io/389-ds-base.git # this way it will create an only .git directory, without files checked out (wait until gitweb will sync)
        # now on your laptop:
        git clone https://git@pagure.io/389-ds-base.git
        cd 389-ds-base
        git remote add $(whoami) git+ssh://git.engineering.redhat.com/srv/git/users/$(whoami)/389-ds-base.git
  2. Go to the cloned directory

  3. Create a new branch for your work:

    • git checkout -b new_test_suite

  4. Check out PEP8 cheat sheet:

  5. Use ./dirsrvtests/create_test.py tool to generate new test.py file. Usage:

    • create_ticket.py -t|--ticket <ticket number> -s|--suite <suite name> [ i|--instances <number of standalone instances> [ -m|--masters <number of masters> -h|--hubs <number of hubs> -c|--consumers <number of consumers> ] -o|--outputfile]
    • Create a test suite script using "-s|--suite" instead of using "-t|–ticket". One day, all 'tickets' will be transferred to 'suites', so try to avoid the 'tickets' and try to find the place in 'suites' for you case. Ask around is you have doubts.
    • Option "-i" can add multiple standalone instances.  However, you can not mix "-i" with the replication options(-m, -h ,-c).
    • For example:
      • create_test.py -s basic -m 2 -o ./dirsrvtests/tests/suites/basic/basic_test.py
      •  # It will create basic_test.py with two masters set up and put the file to right dir
    • If you are creating a test suite, the script will add one test case for you with generated ID in the docstring (and it will check it for uniqueness)
    • Please, add more ID (to new test cases) with the next command and check if it is unique for other tests
      • python -c 'import uuid; print(uuid.uuid4())'
  6. Add some fixture(s), if needed. The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute.
    • For example:

      @pytest.fixture
      def rdn_write_setup(topology_m2):
          topology_m2.ms["master1"].add_s(ENTRY)
          def fin():
              topology_m2.ms["master1"].delete_s(ENTRY_DN)
          request.addfinalizer(fin)
    • It will add some entry to the master1 in the beginning of the test case and delete this entry after test case is finished.
  7. Add test case(s). It should be defined as function which name starts with "test_"
    • For example:
      • def test_search_limits_fail(topology, rdn_write_setup):
    • You can put any amount of created fixtures as the arguments
  8. Write some good code with encapsulations, assertions etc.
  9. Commit and push your code to your repo:

    git add ./dirsrvtests/tests/suites/basic/basic_test.py
    git commit
    git push $(whoami)
  10. Test your script:
    1. Go to the "beaker" branch of dirsrv-tests repo, to the beaker/upstream-tests dir
    2. Run your tests, for debug mode - set DEBUGGING environment variable to something. like - export DEBUGGING=abcd
    3. To disable debug mode - unset it like export DEBUGGING=
    4. If you want to have a closer look, you can log to the machine with choosing "s" option and run tests manually:
      • py.test -v -s /mnt/testarea/test/ds/dirsrvtests/suites/basic
  11. If everything is alright, then create a patch file for a review: 
    1. Go back to ds or lib389 dir (depends on where you want to send the patch) and do:

      git checkout master
      git pull
      git checkout new_test_suite
      git rebase master
      git format-patch -1
    2. Basic guidelines for the commit message format
      • Separate subject from body with a blank line
      • Limit the subject line to 50 characters
      • Capitalizethesubject line
      • Do not end the subject line with a period
      • Use the imperative mood in the subject line
      • Wrap the body at 72 characters
      • Use the body to explain what and why vs. how
      • In the end, put a link to the ticket
      • Add "Reviewed by: ?" line. Example:
        • Issue 48085 - Expandthereplacceptancetest suite

          Description: Add 6 more test cases to the replication test suite
          as a part of the TET to pytest porting initiative.
          Increase the number of seconds we wait before the results check.

          https://pagure.io/389-ds-base/issue/48085

          Reviewed by: ?

  12. Fixing Review Issues
    1. If there are issues with your patch, git allows you to fix your commits.
    2. If you're not already in that branch
    3. git checkout new_test_suite
    4. Make changes to some file
    5. Add changes to your commit and fix the commit message if necessary
      • git commit -a --amend
    6. You can also use “git rebase -i” to “squash” or combine several commits into one commit.

 

A ways to make your code better in a pytest way

Fixtures

Basic info about fixtures - http://pytest.org/latest/fixture.html#fixtures

Scope

  • the scope for which this fixture is shared, one of “function” (default), “class”, “module”, “session”
  • Use “function”, if you want fixture to be applied for every test case where it appears
  • Use “module”, if you want fixture to be applied for a whole test suite (file you run)

Parametrizing

  • Fixture functions can be parametrized in which case they will be called multiple times, each time executing the set of dependent tests, i. e. the tests that depend on this fixture.
  • You should put your params in list and then access it within you fixture with request.paramFor example:

    # First it will test with adding and deleting ENTRY to the first master then to the second
    @pytest.fixture(params=[0, 1])
    def rdn_write_setup(topology_m2):
        m_num = request.param
        topology_m2.ms["master{}".format(m_num)].add_s(ENTRY)
        def fin():
            topology_m2.ms["master{}".format(m_num)].delete_s(ENTRY_DN)
        request.addfinalizer(fin)

Test cases

Parametrizing

  • The built-in pytest.mark.parametrize decorator enables parameterization of arguments for a test function. For example:

    ROOTDSE_DEF_ATTR_LIST = ('namingContexts',
                             'supportedLDAPVersion',
                             'supportedControl',
                             'supportedExtension',
                             'supportedSASLMechanisms',
                             'vendorName',
                             'vendorVersion')
    @pytest.mark.parametrize("rootdse_attr_name", ROOTDSE_DEF_ATTR_LIST)
    def test_def_rootdse_attr(topology_st, import_example_ldif, rootdse_attr_name):
        """Tests that operational attributes
        are not returned by default in rootDSE searches
        """
    
        log.info("Assert rootdse search hasn't {} attr".format(rootdse_attr_name))
        entries = topology_st.standalone.search_s("", ldap.SCOPE_BASE)
        entry = str(entries[0])
        assert rootdse_attr_name not in entry
  • As you can see, unlike the fixture parametrizing, in the test case you should first put the name of attributes, then the list (or tuple) with values, and then put the attribute to the function declaration.

  • You can specify a few attributes for parametrizing

    @pytest.mark.parametrize("test_input,expected", [
        ("3+5", 8),
        ("2+4", 6),
        ("6*9", 42),])
    def test_eval(test_input, expected):
        assert eval(test_input) == expected

Marking test functions and selecting them for a run

  • You can “mark” a test function with custom meta data like this:

    @pytest.mark.ssl
    def test_search_sec_port():
        pass # perform some search through sec port
  • You can also set a module level marker in which case it will be applied to all functions and methods defined in the module:

    import pytest
    pytestmark = pytest.mark.ssl
  • You can then restrict a test run to only run tests marked with ssl:
    • py.test -v -m ssl
  • Or the inverse, running all tests except the ssl ones:
    • py.test -v -m "not ssl"
  • Select tests based on their node ID
    • You can provide one or more node IDs as positional arguments to select only specified tests. This makes it easy to select tests based on their module, class, method, or function name:
    • py.test -v test_server.py::test_function1 test_server.py::test_function2
  • Use -k expr to select tests based on their name
    • You can use the -k command line option to specify an expression which implements a substring match on the test names instead of the exact match on markers that -m provides. This makes it easy to select tests based on their names
      • py.test -v -k search

      • py.test -v -k "search or modify"

      • py.test -v -k "not modify"

Asserting

  • pytest allows you to use the standard python assert for verifying expectations and values in Python tests. For example, you can write the following:
    def f():
        return 3
    def test_function():
        assert f() == 4
  • You can put the message to assert, it will be shown when error appears:
    • assert a % 2 == 0, "value was odd, should be even"
  • In order to write assertions about raised exceptions, you can use pytest.raises as a context manager like this:

    import pytest
    def test_zero_division():
        with pytest.raises(ZeroDivisionError):
            1 / 0
  • Or even like this, if you expect some particular exception:

    def test_recursion_depth():
        with pytest.raises(RuntimeError) as excinfo:
            def f():
                f()
            f()
        assert 'maximum recursion' in str(excinfo.value)

lib389 and python-ldap functions

Constants

Basic constants

  • DEFAULT_SUFFIX: is set to “dc=example,dc=com

  • DN_DM = "cn=Directory Manager"

  • PW_DM = "password"

  • DN_CONFIG = "cn=config"

  • DN_SCHEMA = "cn=schema"

  • DN_LDBM = "cn=ldbm database,cn=plugins,cn=config"

  • DN_CONFIG_LDBM = "cn=config,cn=ldbm database,cn=plugins,cn=config"

  • DN_USERROOT_LDBM = "cn=userRoot,cn=ldbm database,cn=plugins,cn=config"

  • DN_MONITOR = "cn=monitor"

  • DN_MONITOR_SNMP = "cn=snmp,cn=monitor"

  • DN_MONITOR_LDBM = "cn=monitor,cn=ldbm database,cn=plugins,cn=config"

  • CMD_PATH_SETUP_DS = "setup-ds.pl"

  • CMD_PATH_REMOVE_DS = "remove-ds.pl"

  • CMD_PATH_SETUP_DS_ADMIN = "setup-ds-admin.pl"

  • CMD_PATH_REMOVE_DS_ADMIN = "remove-ds-admin.pl"

For more info check the source code at https://pagure.io/lib389/blob/master/f/lib389/_constants.py . If you need a constant, use this kind of import, do not import all of them:

from lib389._constants import CONSTANT_YOU_NEED
from lib389._constants import (CONSTANT_YOU_NEED_1, CONSTANT_YOU_NEED_2, CONSTANT_YOU_NEED_3,
                               CONSTANT_YOU_NEED_4, CONSTANT_YOU_NEED_5, CONSTANT_YOU_NEED_6)

Add, Modify, and Delete Operations

Please, use these methods for the operations that can't be performed by DSLdapObjects.

# Add an entry
USER_DN = 'cn=mreynolds,{}'.format(DEFAULT_SUFFIX)
standalone.add_s(Entry((USER_DN, {
                              'objectclass': 'top person'.split(),
                              'cn': 'mreynolds',
                              'sn': 'reynolds',
                              'userpassword': 'password'
                          })))

# Modify an entry
standalone.modify_s(USER_DN, [(ldap.MOD_REPLACE, 'cn', 'Mark Reynolds')])

# Delete an entry
standalone.delete_s(USER_DN)

Search and Bind Operations

  • By default when an instance is created and opened, it is already authenticated as the Root DN(Directory Manager).
  • So you can just start searching without having to “bind”

    # Search
    entries = standalone.search_s(DEFAULT_SUFFIX, ldap.SCOPE_SUBTREE, '(cn=*)', ['cn'])
    for entry in entries:
        if 'Mark Reynolds' in entry.data['cn']:
            log.info('Search found "Mark"')
            print(entry.data['cn'])
    
    # Anonymous bind
    bind_dn = ""
    bind_pwd = ""
    
    # Bind as our test entry
    bind_dn = USER_DN
    bind_pwd = "password"
    
    # Bind as Directory Manager
    bind_dn = DN_DM
    bind_pwd = 1
    
    standalone.simple_bind_s(bind_dn, bind_pwd)

Basic instance operations

# First, create a new “instance” of a “DirSrv” object
standalone = DirSrv(verbose=False)
 
# Set up the instance arguments (note - args_instance is a global dictionary
# in lib389, it contains other default values)
args_instance[SER_HOST] = HOST_STANDALONE
args_instance[SER_PORT] = PORT_STANDALONE
args_instance[SER_SERVERID_PROP] = SERVERID_STANDALONE
args_instance[SER_CREATION_SUFFIX] = DEFAULT_SUFFIX
args_standalone = args_instance.copy()
# Allocate the instance - initialize the “DirSrv” object with our arguments
standalone.allocate(args_standalone)
# Check if the instance with the args exists
assert not standalone.exists() 
# Create the instance - this runs setup-ds.pl and starts the server
standalone.create()

# Open the instance - create a connection to the instance,
# and authenticates as the Root DN (cn=directory manager)
standalone.open()
# Done, you can start using the new instance
# While working with DirSrv object, you can set 'verbose' parameter to True in any moment
standalone.verbose = True
# To remove an instance, simply use:
standalone.delete()
# Start, Stop, and Restart the Server
standalone.start(timeout=10)
standalone.stop(timeout=10)
standalone.restart(timeout=10)
 
# Returns True, if the instance was shutdowned disorderly
standalone.detectDisorderlyShutdown()

Setting up SSL/TLS

from lib389._constants import DEFAULT_SUFFIX, SECUREPORT_STANDALONE1

standalone.stop()
 
# Re-init (create) the nss db
# pin.txt is created here and the password randomly generated
assert(standalone.nss_ssl.reinit() is True)
 
# Create a self signed CA
# noise.txt is created here
assert(standalone.nss_ssl.create_rsa_ca() is True)
 
# Create a key and a cert that is signed by the self signed ca
# This will use the hostname from the DS instance, and takes a list of extra names to take.
assert(standalone.nss_ssl.create_rsa_key_and_cert() is True)
    
standalone.start()

# Create "cn=RSA,cn=encryption,cn=config" with next properties:
# {'cn': 'RSA', 'nsSSLPersonalitySSL': 'Server-Cert', 'nsSSLActivation': 'on', 'nsSSLToken': 'internal (software)'}
standalone.rsa.create()
# Set the secure port and nsslapd-security
standalone.config.set('nsslapd-secureport', str(SECUREPORT_STANDALONE1))
standalone.config.set('nsslapd-security', 'on')
standalone.sslport = SECUREPORT_STANDALONE1

# Restart to allow certmaps to be re-read: Note, we CAN NOT use post_open
standalone.restart(post_open=False)

Certification-based authentication

You need to setup and turn on SSL first (use the previous chapter).

from lib389.config import CertmapLegacy

standalone.stop()
 
# Create a user
assert(standalone.nss_ssl.create_rsa_user('testuser') is True)
 
# Get the details of where the key and crt are
#  {'ca': ca_path, 'key': key_path, 'crt': crt_path}
tls_locs = standalone.nss_ssl.get_rsa_user('testuser')

standalone.start()

# Create user in the directory 
users = UserAccounts(standalone, DEFAULT_SUFFIX)
users.create(properties={
        'uid': 'testuser',
        'cn' : 'testuser',
        'sn' : 'user',
        'uidNumber' : '1000',
        'gidNumber' : '2000',
        'homeDirectory' : '/home/testuser'
})

# Turn on the certmap
cm = CertmapLegacy(standalone)
certmaps = cm.list()
certmaps['default']['DNComps'] = ''
certmaps['default']['FilterComps'] = ['cn']
certmaps['default']['VerifyCert'] = 'off'
cm.set(certmaps)

# Restart to allow certmaps to be re-read: Note, we CAN NOT use post_open
standalone.restart(post_open=False)

# Now attempt a bind with TLS external
conn = standalone.openConnection(saslmethod='EXTERNAL', connOnly=True, certdir=standalone.get_cert_dir(), userkey=tls_locs['key'], usercert=tls_locs['crt'])

assert(conn.whoami_s() == "dn: uid=testuser,ou=People,dc=example,dc=com")

Replication

Basic configuration

  • After the instance is created, you can enable it for replication and set up a replication agreement.

    from lib389.replica import Replicas
     
    # Enable replication 
    replicas = Replicas(standalone)
    replica = replicas.enable(suffix=DEFAULT_SUFFIX,
                              role=REPLICAROLE_MASTER,
                              replicaID=REPLICAID_MASTER_1)
    # Set up replication agreement properties
    properties = {RA_NAME:           r'meTo_{}:{}'.format(master2.host, port=master2.port),
                  RA_BINDDN:         defaultProperties[REPLICATION_BIND_DN],
                  RA_BINDPW:         defaultProperties[REPLICATION_BIND_PW],
                  RA_METHOD:         defaultProperties[REPLICATION_BIND_METHOD],
                  RA_TRANSPORT_PROT: defaultProperties[REPLICATION_TRANSPORT]}
    
    # Create the agreement
    repl_agreement = standalone.agreement.create(suffix=DEFAULT_SUFFIX, 
                                                 host=master2.host,
                                                 port=master2.port,
                                                 properties=properties)
    # “master2” refers to another, already created, DirSrv instance(like “standalone”)
    # “repl_agreement” is the “DN” of the newly created agreement - this DN is needed later to do certain tasks
    
    # Initialize the agreement, wait for it complete, and test that replication is really working
    standalone.agreement.init(DEFAULT_SUFFIX, master2.host, master2.port)
    standalone.waitForReplInit(repl_agreement)
    assert standalone.testReplication(DEFAULT_SUFFIX, master2)

Agreements

# Create
properties = {RA_NAME: ('meTo_{}:{}'.format(topology.consumer.host,
                                            topology.consumer.port)),
              RA_BINDDN: defaultProperties[REPLICATION_BIND_DN],
              RA_BINDPW: defaultProperties[REPLICATION_BIND_PW],
              RA_METHOD: defaultProperties[REPLICATION_BIND_METHOD],
              RA_TRANSPORT_PROT: defaultProperties[REPLICATION_TRANSPORT]}
# repl_agreement contains a DN of the entry
repl_agreement = master1.agreement.create(suffix=DEFAULT_SUFFIX,
                                          host=topology.consumer.host,
                                          port=topology.consumer.port,
                                          properties=properties)
 
# List
# With agreement DN
ents = master1.agreement.list(agmtdn=repl_agreement)
# With suffix
ents = master1.agreement.list(suffix=DEFAULT_SUFFIX)
# With suffix, host, and port
ents = master1.agreement.list(suffix=DEFAULT_SUFFIX,
                              consumer_host=topology.consumer.host,
                              consumer_port=topology.consumer.port)
 
# Delete
# With agreement DN
ents = master1.agreement.delete(agmtdn=repl_agreement)
# With suffix
ents = master1.agreement.delete(suffix=DEFAULT_SUFFIX)
# With suffix, host, and port
ents = master1.agreement.delete(suffix=DEFAULT_SUFFIX,
                                consumer_host=topology.consumer.host,
                                consumer_port=topology.consumer.port)
 
# Init
# Trigger a total update of the consumer replica
ents = master1.agreement.init(suffix=DEFAULT_SUFFIX,
                              consumer_host=topology.consumer.host,
                              consumer_port=topology.consumer.port)
 
# Pause
ents = master1.agreement.pause(agmtdn=repl_agreement)
 
# Resume
ents = master1.agreement.resume(agmtdn=repl_agreement)
 
# Schedule
ents = master1.agreement.schedule(agmtdn=repl_agreement, interval="0000-1234 6420")
# Predefined options for interval: Agreement.ALWAYS, Agreement.NEVER
 
# Changes
# Return a list of changes sent by this agreement
ents = master1.agreement.changes(agmtdn=repl_agreement)

Changelog

# Create
changelog_dn = standalone.changelog.create() # Additionaly you can specify 'dbname=' (default is DEFAULT_CHANGELOG_DB)
 
# List
changelog_entries = standalone.changelog.list(changelogdn=changelog_dn)
 
# Delete
standalone.changelog.delete()

Replication tools

from lib389.repltools import ReplTools
 
# Gather all the CSN strings from the access and verify all of those CSNs exist on all the other replicas.
# dirsrv_replicas - a list of DirSrv objects.  The list must begin with master replicas
# ignoreCSNs - an optional string of csns to be ignored
# if the caller knows that some csns can differ eg.: '57e39e72000000020000|vucsn-57e39e76000000030000'
ReplTools.checkCSNs([master1, master2], ignoreCSNs=None)
 
# Find and measure the convergence of entries from a replica, and
# print a report on how fast all the "ops" replicated to the other replicas.
# suffix - Replicated suffix
# ops - A list of "operations" to search for in the access logs
# replica - Dirsrv object where the entries originated
# all_replicas - A list of Dirsrv replicas
# It returns - The longest time in seconds for an operation to fully converge
longest_time = ReplTools.replConvReport(DEFAULT_SUFFIX, ops, master1, [master1, master2])

# Take a list of DirSrv Objects and check to see if all of the present
# replication agreements are idle for a particular backend
assert(ReplTools.replIdle([master1, master2], suffix=DEFAULT_SUFFIX))
defaultProperties = {
            REPLICATION_BIND_DN: "cn=replrepl,cn=config",
            REPLICATION_BIND_PW

# Create an entry that will be used to bind as replication manager
ReplTools.createReplManager(standalone,
                            repl_manager_dn=defaultProperties[REPLICATION_BIND_DN],
                            repl_manager_pw=defaultProperties[REPLICATION_BIND_PW])

DSLdapObject operations

  • DSLdapObjects and DSLdapObject are inherited by other objects like Replicas, Tasks, MappingTrees, etc.
  • DSLdapObjects represent the next idea: "Everything is an instance of something that exists in this way", i.e. we unite LDAP entries by some set of parameters with the object. For instance, UserAccounts(standalone, DEFAULT_SUFFIX) represents all user accounts under DEFAULT_SUFFIX.
  • With it, we can list, create and get entries.

    # We can define the new object
    users = UserAccounts(standalone, DEFAULT_SUFFIX)
    backends = Backends(standalone)
     
    # List - return a list object with instances of DSLdapObject in it 
    for backend in backends.list():
        # here, backend is an instance of DSLdapObject
        print(backend.display())
     
    # Create one DSLdapObject
    user_properties = {
        'uid': 'testuser',
        'cn': 'testuser',
        'sn': 'user',
        'uidNumber': '1000',
        'gidNumber': '2000',
        'homeDirectory': '/home/testuser'
    }
    # Other parameters are carried out - it takes 'rdn' from properties, and 'basedn' from parent class
    user = users.create(properties=user_properties)
     
    # Get one DSLdapObject (that already exists in the directory)
    # with 'selector'
    user = users.get('testuser')
    # or with 'dn'
    user = users.get(dn='uid=testuser,{}'.format(DEFAULT_SUFFIX))
  • After obtaining the entry we start to work with DSLdapObject instance.

    # Bind
    # If the account can be bound to, this will attempt to do so.
    # We don't check for exceptions, just pass them back!
    user.bind('password')
     
    user.delete()
    
    # Get LDAP entry object ('Entry' object)
    user_entry = user.raw_entry()
    
    # The same as previous, but represent it as a string LDIF
    user_ldif = user.display()
     
    # Get attrs
    user_all_attrs = user.get_all_attrs() # real attributes + operational attributes
     
    # Use one of these functions with exact return type you need
    user_attr_val_s = user.get_attr_val_utf8('stringAttr')
    user_attr_vals_s = user.get_attr_vals_utf8('stringAttr')
    user_attr_val_i = user.get_attr_val_int('numAttr')
    user_attr_vals_i = user.get_attr_vals_int('numAttr')
    user_attr_val_b = user.get_attr_val_bytes('byteAttr')
    user_attr_vals_b = user.get_attr_vals_bytes('byteAttr')
    user_home = user.display_attr('homeDirectory') # Get all values of given attribute - 'attr: value\n'
    
    # Get dn
    user_dn = user.dn()
    
    # Get rdn
    user_rdn = user.rdn()
     
    # Check if some attr, or some attr / value exist on the entry.
    assert(user.present(attr='homeDirectory'))
    assert(user.present(attr='homeDirectory', value='/home/testuser'))
     
    # Set - more general method for 'modify_s' operations (default action=ldap.MOD_REPLACE)
    user.set('homeDirectory', '/home/testuser_new')
     
    user.replace('homeDirectory', '/home/testuser')
    
    user.remove('homeDirectory', '/home/testuser')
    user.remove_all('homeDirectory')
    
    # Multiple modification - the same as set, but accepts list of tuples of [(action, key, value),]
    mods = [('homeDirectory', '/home/testuser_new', ldap.MOD_DELETE),
            ('uidNumber', '3000', ldap.MOD_REPLACE),
            ('gidNumber', '3000', ldap.MOD_REPLACE)]
    user.apply_mods(mods):
    
    # Compare if two RDN objects have same attributes and values.
    # This comparison is a loose comparison, not a strict one i.e. "this object *is* this other object"
    # It will just check if the attributes are same.
    # 'nsUniqueId' attribute is not checked intentionally because we want to compare arbitrary objects
    # i.e they may have different 'nsUniqueId' but same attributes.
    # Example:
    #   cn=user1,ou=a
    #   cn=user1,ou=b
    # Comparision of these two objects should result in same, even though their 'nsUniqueId' attribute differs.
    # This function returns 'True' if objects have same attributes else returns 'False'
    assert(UserAccount.compare(testuser1, testuser2) == False)

Monitor

# Monitor and MonitorLDBM are the simple DSLdapObject things.
# You can use all methods from chapter above to get current server performance detail
version = standalone.monitor.get_attr_val('version')
dbcachehit = standalone.monitorldbm.get_attr_val('dbcachehit')

Replicas

from lib389.replica import Replicas
 
replicas = Replicas(standalone)
# Enable replication
# - changelog will be created
# - replica manager will be with the defaults
# - replica.create() will be executed
replica = replicas.enable(suffix=DEFAULT_SUFFIX,
                          role=REPLICAROLE_MASTER,
                          replicaID=REPLICAID_MASTER_1)
# Roles - REPLICAROLE_MASTER, REPLICAROLE_HUB, and REPLICAROLE_CONSUMER
# For masters and hubs you can use the constants REPLICAID_MASTER_X and REPLICAID_HUB_X
# Change X for a number from 1 to 100 - for role REPLICAROLE_MASTER only

# Disable replication
# - agreements and replica entry will be deleted
# - changelog is not deleted (but should?)
replicas.disable(suffix=DEFAULT_SUFFIX)
 
# Get RUV entry
replicas.get_ruv_entry()
 
# Get DN 
replicas.get_dn(suffix)

# Promote
replicas.promote(suffix=DEFAULT_SUFFIX,
                 newrole=REPLICAROLE_MASTER,
                 binddn=REPL_BINDDN,
                 rid=REPLICAID_MASTER_1)
# Demote
replicas.demote(suffix=DEFAULT_SUFFIX,
                newrole=REPLICAROLE_CONSUMER)
# Test, that replication works
replicas.test(master2)

# Additional replica object methods
# Get role
replica.get_role()
 
replica.deleteAgreements()

Backends

from lib389.backend import Backends
 
backends = Backends(standalone)
backend = backends.create(properties={BACKEND_SUFFIX: 'o=new_suffix', # mandatory
                                      BACKEND_NAME: new_backend,      # mandatory
                                      BACKEND_SAMPLE_ENTRIES: '001003006'})

# Properties you can specify:
# BACKEND_NAME - 'somename'
# BACKEND_READONLY - 'on' | 'off'
# BACKEND_REQ_INDEX - 'on' | 'off'
# BACKEND_CACHE_ENTRIES - 1 to (2^32 - 1) on 32-bit systems or (2^63 - 1) on 64-bit systems or -1, which means limitless
# BACKEND_CACHE_SIZE - 500 kilobytes to (2^32 - 1) on 32-bit systems and to (2^63 - 1) on 64-bit systems
# BACKEND_DNCACHE_SIZE - 500 kilobytes to (2^32 - 1) on 32-bit systems and to (2^63 - 1) on 64-bit systems
# BACKEND_DIRECTORY - Any valid path to the database instance
# BACKEND_CHAIN_BIND_DN - DN of the multiplexor
# BACKEND_CHAIN_BIND_PW - password of the multiplexor
# BACKEND_CHAIN_URLS - Any valid remote server LDAP URL
# BACKEND_SUFFIX - 'o=somesuffix'
# BACKEND_SAMPLE_ENTRIES - version of confir i.e. '001003006'
 
backend.delete()
 
# Create sample entries
backend.create_sample_entries(version='001003006')

Domain

# After the creating a backend, sometimes you don't need a lot of entries under the created suffix
# So instead of using BACKEND_SAMPLE_ENTRIES you can create simple domain entry using the next object:
from lib389.idm.domain import Domain
domain = Domain(standalone], 'dc=test,dc=com')
domain.create(properties={'dc': 'test', 'description': 'dc=test,dc=com'})
 
# It will be deleted with the 'backend.delete()'

Mapping trees

# In the majority of test cases, it is better to use 'Backends' for the operation,
# because it creates the mapping tree for you. Though if you need to create a mapping tree, you can.
# Just work with it as with usual DSLdapObject and DSLdapObjects
# For instance:
from lib389.mappingTree import MappingTrees
mts = MappingTrees(standalone)
mt = mts.create(properties={
        'cn': ["dc=newexample,dc=com",],
        'nsslapd-state' : 'backend',
        'nsslapd-backend' : 'someRoot',
        })
# It will be deleted with the 'backend.delete()'


UserAccounts

# There is a basic way to work with it
from lib389.idm.user import UserAccounts
users = UserAccounts(standalone, DEFAULT_SUFFIX)
user_properties = {
       'uid': USER_NAME,
       'cn' : USER_NAME,
       'sn' : USER_NAME,
       'userpassword' : USER_PWD,
       'uidNumber' : '1000',
       'gidNumber' : '2000',1
       'homeDirectory' : '/home/{}'.format(USER_NAME)
        }
testuser = users.create(properties=user_properties)

# After this you can:
# Get the list of them
users.list()

# Get some user:
testuser = users.get('testuser')
# or
testuser = users.list()[0] # You can loop through 'for user in users:'

# Set some attribute to the entry
testuser.set('userPassword', 'password')

# Bind as the user
conn = testuser.bind('password') # It will create a new connection
conn.modify_s()
conn.unbind_s()

# Delete
testuser.delete()

 

Groups

# Group and Groups additionaly have 'is_member', 'add_member' and 'remove_member' methods
# PosixGroup and PosixGroups have 'check_member' and 'add_member'
from lib389.idm.group import Groups
from lib389.idm.posixgroup import PosixGroups

groups = Groups(standalone, DEFAULT_SUFFIX)
posix_groups = PosixGroups(standalone, DEFAULT_SUFFIX)
group_properties = {
       'cn' : 'group1',
       'description' : 'testgroup'
       }
group = groups.create(properties=group_properties)

# So now you can:
# Check the membership - shouldn't we make it consistent?
assert(not group.is_member(testuser.dn))
assert(not posix_groups.check_member(testuser.dn))

group.add_member(testuser.dn)
posix_groups.add_member(testuser.dn)

# Remove member - add the method to PosixGroups too?
group.remove_member(testuser.dn)

group.delete()

Services and Organisational Units

# Don't forget that Services requires created rdn='ou=Services'
# This you can create with OrganisationalUnits
 
from lib389.idm.organisationalunit import OrganisationalUnits
from lib389.idm.services import ServiceAccounts
 
ous = OrganisationalUnits(standalone, DEFAULT_SUFFIX)
services = ServiceAccounts(standalone, DEFAULT_SUFFIX)

# Create the OU for them
ous.create(properties={
        'ou': 'Services',
        'description': 'Computer Service accounts which request DS bind',
    })

# Now, we can create the services from here.
service = services.create(properties={
    'cn': 'testbind',
    'userPassword': 'Password1'
    })

conn = service.bind('Password1')
conn.unbind_s()

Indexes

from lib389.index import Indexes
 
indexes = Indexes(standalone)
 
# create and delete a default index.
index = indexes.create(properties={
    'cn': 'modifytimestamp',
    'nsSystemIndex': 'false',
    'nsIndexType': 'eq'
    })

default_index_list = indexes.list()
found = False
for i in default_index_list:
    if i.dn.startswith('cn=modifytimestamp'):
        found = True
assert found
index.delete()

default_index_list = indexes.list()
found = False
for i in default_index_list:
    if i.dn.startswith('cn=modifytimestamp'):
        found = True
assert not found

Tasks

Besides the predefined tasks (which described in a chapter below) you can create your own task objects with specifying a DN (https://pagure.io/lib389/blob/master/f/lib389/_constants.py#_134)

from lib389.tasks import Task
 
newtask = Task(instance, dn) # Should we create Tasks and put the precious to TasksLegacy?

newtask.create(rdn, properties, basedn)
 
# Check if the task is complete
assert(newtask.is_complete())

# Check task's exit code if task is complete, else None
if newtask.is_complete():
    exit_code = newtask.get_exit_code()

# Wait until task is complete
newtask.wait()
 
# If True,  waits for the completion of the task before to return
args = {TASK_WAIT: True}
 
# Some tasks ca be found only under old object. You can access them with this:
standalone.tasks.importLDIF(DEFAULT_SUFFIX, path_ro_ldif, args)
standalone.tasks.exportLDIF(DEFAULT_SUFFIX, benamebase=None, output_file=path_to_ldif, args)
standalone.tasks.db2bak(backup_dir, args)
standalone.tasks.bak2db(bename=None, backup_dir, args)
standalone.tasks.reindex(suffix=None, benamebase=None, attrname=None, args)
standalone.tasks.fixupMemberOf(suffix=None, benamebase=None, filt=None, args)
standalone.tasks.fixupTombstones(bename=None, args)
standalone.tasks.automemberRebuild(suffix=DEFAULT_SUFFIX, scope='sub', filterstr='objectclass=top', args)
standalone.tasks.automemberExport(suffix=DEFAULT_SUFFIX, scope='sub', fstr='objectclass=top', ldif_out=None, args)
standalone.tasks.automemberMap(ldif_in=None, ldif_out=None, args)
standalone.tasks.fixupLinkedAttrs(linkdn=None, args)
standalone.tasks.schemaReload(schemadir=None, args)
standalone.tasks.fixupWinsyncMembers(suffix=DEFAULT_SUFFIX, fstr='objectclass=top', args)
standalone.tasks.syntaxValidate(suffix=DEFAULT_SUFFIX, fstr='objectclass=top', args)
standalone.tasks.usnTombstoneCleanup(suffix=DEFAULT_SUFFIX, bename=None, maxusn_to_delete=None, args)
standalone.tasks.sysconfigReload(configfile=None, logchanges=None, args)
standalone.tasks.cleanAllRUV(suffix=None, replicaid=None, force=None, args)
standalone.tasks.abortCleanAllRUV(suffix=None, replicaid=None, certify=None, args)
standalone.tasks.upgradeDB(nsArchiveDir=None, nsDatabaseType=None, nsForceToReindex=None, args)

Plugins

You can take plugin constant names here - https://pagure.io/lib389/blob/master/f/lib389/_constants.py#_164  

# Plugin and Plugins additionaly have 'enable', 'disable' and 'status' methods
# Here I show you basic way to work with it. Additional methods of complex plugins will be described in subchapters
 
from lib389.plugin import Plugins, ACLPlugin
from lib389._constants import PLUGIN_ACL
 
# You can just enable/disable plugins from Plugins interface
plugins = Plugins(standalone)
plugins.enable(PLUGIN_ACL) 
 
# Or you can first 'get' it and then work with it (make sense if your plugin is a complex one)
aclplugin = ACLPlugin(standalone)

aclplugin.enable()

aclplugin.disable()

# True if nsslapd-pluginEnabled is 'on', False otherwise - change the name?
assert(uniqplugin.status())

In the chapters below, you can find the plugins that have additional wrapper methods. If the plugin you looking for is not there, use basic DSLdapObject and Plugin methods.

MemberOf plugin

from lib389.plugin import MemberOfPlugin
 
memberofplugin = MemberOfPlugin(standalone)

# Create fixup task and return the object
memberof_task = memberofplugin.fixup(basedn, _filter=None)

USN

from lib389.plugin import USNPlugin
 
usnplugin = USNPlugin(standalone)

# Check if global mode is set
assert(usnplugin.is_global_mode_set())
 
# Set 'nsslapd-entryusn-global' to 'on'
usnplugin.enable_global_mode()

# Set 'nsslapd-entryusn-global' to 'off'
usnplugin.disable_global_mode()

# Create USN tombstone cleanup task and return the object
# Optionaly you can specify 'suffix', 'backend' and 'maxusn_to_delete'
# It is mandatory to specify either 'suffix' or 'backend'
usnplugin.cleanup(suffix=None, backend=None, max_usn=None)

RootDSE

# Get attribute values of 'supportedSASLMechanisms'
standalone.rootdse.supported_sasl()

# Returns True or False
assert(standalone.rootdse.supports_sasl_gssapi()
assert(standalone.rootdse.supports_sasl_ldapssotoken()
assert(standalone.rootdse.supports_sasl_plain()
assert(standalone.rootdse.supports_sasl_external()
assert(standalone.rootdse.supports_exop_whoami()
assert(standalone.rootdse.supports_exop_ldapssotoken_request()
assert(standalone.rootdse.supports_exop_ldapssotoken_revoke()

ACI operations

  • Add ACI

    ACI_TARGET = ('(targetfilter ="(ou=groups)")(targetattr ="uniqueMember '
                  '|| member")')
    ACI_ALLOW = ('(version 3.0; acl "Allow test aci";allow (read, search, '
                 'write)')
    ACI_SUBJECT = ('(userdn="ldap:///dc=example,dc=com??sub?(ou=engineering)" '
                   'and userdn="ldap:///dc=example,dc=com??sub?(manager=uid='
                   'wbrown,ou=managers,dc=example,dc=com) || ldap:///dc=examp'
                   'le,dc=com??sub?(manager=uid=tbrown,ou=managers,dc=exampl'
                   'e,dc=com)" );)')
     
    group_dn = 'cn=testgroup,{}'.format(DEFAULT_SUFFIX)
    gentry = Entry(group_dn)
    gentry.setValues('objectclass', 'top', 'extensibleobject')
    gentry.setValues('cn', 'testgroup')
    gentry.setValues('aci', ACI_BODY)
    standalone.add_s(gentry)
    # The same you can do with modify_s, just use ACI_BODY
  • Get and parse ACI

    acis = standalone.aci.list()
    aci = acis[0]
    
    assert aci.acidata == {
        'allow': [{'values': ['read', 'search', 'write']}],
        'target': [], 'targetattr': [{'values': ['uniqueMember', 'member'],
                                      'equal': True}],
        'targattrfilters': [],
        'deny': [],
        'acl': [{'values': ['Allow test aci']}],
        'deny_raw_bindrules': [],
        'targetattrfilters': [],
        'allow_raw_bindrules': [{'values': [(
            'userdn="ldap:///dc=example,dc=com??sub?(ou=engineering)" and'
            ' userdn="ldap:///dc=example,dc=com??sub?(manager=uid=wbrown,'
            'ou=managers,dc=example,dc=com) || ldap:///dc=example,dc=com'
            '??sub?(manager=uid=tbrown,ou=managers,dc=example,dc=com)" ')]}],
        'targetfilter': [{'values': ['(ou=groups)'], 'equal': True}],
        'targetscope': [],
        'version 3.0;': [],
        'rawaci': complex_aci
    }
     
    # You can get a raw ACI
    raw_aci = aci.getRawAci()

Parsing logs

  • lib389 has a nice module to work with logs. You can:

    # Get array of all lines (including rotated and compresed logs):
    standalone.ds_access_log.readlines_archive()
    standalone.ds_error_log.readlines_archive()
    # Get array of all lines (without rotated and compresed logs):
    standalone.ds_access_log.readlines()
    standalone.ds_error_log.readlines()
    # Get array of lines that match the regex pattern:
    standalone.ds_access_log.match_archive('.*fd=.*')
    standalone.ds_error_log.match_archive('.*fd=.*')
    standalone.ds_access_log.match('.*fd=.*')
    standalone.ds_error_log.match('.*fd=.*')
    # Break up the log line into the specific fields:
    assert(standalone.ds_error_log.parse_line('[27/Apr/2016:13:46:35.775670167 +1000]     slapd started.  Listening on All Interfaces port 54321 for LDAP requests') == {'timestamp': '[27/Apr/2016:13:46:35.775670167 +1000]', 'message': 'slapd starte    d.  Listening on All Interfaces port 54321 for LDAP requests', 'datetime': datetime.datetime(2016, 4, 27, 13, 0, 0, 775670, tzinfo=tzoffset(No    ne, 36000))})

Setting up a config

# Set config attribute:
standalone.config.set('passwordStorageScheme', 'SSHA')
 
# Reset config attribute (by deleting it):
standalone.config.reset('passwordStorageScheme')

# Enable/disable logs (error, access, audit):
standalone.config.enable_log('error')
standalone.config.disable_log('access')

# Set loglevel for errors log.
# If 'update' set to True, it will add the 'vals' to existing values in the loglevel attribute 
standalone.config.loglevel(vals=(LOG_DEFAULT,), service='error', update=False)
# You can get log levels from lib389._constants
(LOG_TRACE,
 LOG_TRACE_PACKETS,
 LOG_TRACE_HEAVY,
 LOG_CONNECT,
 LOG_PACKET,
 LOG_SEARCH_FILTER,
 LOG_CONFIG_PARSER,
 LOG_ACL,
 LOG_ENTRY_PARSER,
 LOG_HOUSEKEEPING,
 LOG_REPLICA,
 LOG_DEFAULT,
 LOG_CACHE,
 LOG_PLUGIN,
 LOG_MICROSECONDS,
 LOG_ACL_SUMMARY) = [1 << x for x in (list(range(8)) + list(range(11, 19)))]
 
# Set 'nsslapd-accesslog-logbuffering' to 'on' if True, otherwise set it to 'off'
standalone.config.logbuffering(True)

 

DSEldif

from lib389.dseldif import DSEldif
 
dse_ldif = DSEldif(topo.standalone)
 
# Get a list of attribute values under a given entry
config_cn = dse_ldif.get(DN_CONFIG, 'cn')
 
# Add an attribute under a given entry
dse_ldif.add(DN_CONFIG, 'someattr', 'someattr_value')
 
# Replace attribute values with a new one under a given entry. It will remove all previous 'someattr' values
dse_ldif.replace(DN_CONFIG, 'someattr', 'someattr_value')

# Delete attributes under a given entry
dse_ldif.delete(DN_CONFIG, 'someattr')
dse_ldif.delete(DN_CONFIG, 'someattr', 'someattr_value')

Ldclt

This class will allow general usage of ldcltIt's not meant to expose all the functions. Just use ldclt for that.

# Creates users as user<min through max>. Password will be set to password<number>
# This will automatically work with the bind loadtest.
# Template
# objectClass: top
# objectclass: person
# objectClass: organizationalPerson
# objectClass: inetorgperson
# objectClass: posixAccount
# objectClass: shadowAccount
# sn: user[A]
# cn: user[A]
# givenName: user[A]
# description: description [A]
# userPassword: user[A]
# mail: user[A]@example.com
# uidNumber: 1[A]
# gidNumber: 2[A]
# shadowMin: 0
# shadowMax: 99999
# shadowInactive: 30
# shadowWarning: 7
# homeDirectory: /home/user[A]
# loginShell: /bin/false
topology.instance.ldclt.create_users('ou=People,{}'.format(DEFAULT_SUFFIX), max=1999)

# Run the load test for a few rounds
topology.instance.ldclt.bind_loadtest('ou=People,{}'.format(DEFAULT_SUFFIX), max=1999)

Password

from lib389.passwd import password_hash, password_generate
 
bindir = standalone.ds_paths.bin_dir
PWSCHEMES = [
    'SHA1',
    'SHA256',
    'SHA512',
    'SSHA',
    'SSHA256',
    'SSHA512',
    'PBKDF2_SHA256',
]
 
# Generate password
raw_secure_password = password_generate()
 
# Encrypt the password
# default scheme is 'SSHA512'
secure_password = password_hash(raw_secure_password, scheme='SSHA256', bin_dir=bindir)

Paths

# You can get any variable from the list bellow. Like this:
product = standalone.ds_paths.product
 
variables = [
    'product',
    'version',
    'user',
    'group',
    'root_dn',
    'prefix',
    'bin_dir',
    'sbin_dir',
    'lib_dir',
    'data_dir',
    'tmp_dir',
    'sysconf_dir',
    'config_dir',
    'schema_dir',
    'cert_dir',
    'local_state_dir',
    'run_dir',
    'lock_dir',
    'log_dir',
    'inst_dir',
    'db_dir',
    'backup_dir',
    'ldif_dir',
    'initconfig_dir',
]

Schema

# Get the schema as an LDAP entry
schema = standalone.schema.get_entry()
 
# Get the schema as a python-ldap SubSchema object
subschema = standalone.schema.get_subschema()
 
# Get a list of the schema files in the instance schemadir
schema_files = standalone.schema.list_files()


# Convert the given schema file name to its python-ldap format suitable for passing to ldap.schema.SubSchema()
parsed = standalone.schema.file_to_ldap('/full/path/to/file.ldif')
 
# Convert the given schema file name to its python-ldap format ldap.schema.SubSchema object
parsed = standalone.schema.file_to_subschema('/full/path/to/file.ldif')
 
# Add a schema element to the schema
standalone.schema.add_schema(attr, val)
 
# Delete a schema element from the schema
standalone.schema.del_schema(attr, val)
 
# Add 'attributeTypes' definition to the schema 
standalone.schema.add_attribute(attributes)

# Add 'objectClasses' definition to the schema
standalone.schema.add_objectclass(objectclasses)

# Get a schema nsSchemaCSN attribute
schema_csn = standalone.schema.get_schema_csn()

# Get a list of ldap.schema.models.ObjectClass objects for all objectClasses supported by this instance
objectclasses = standalone.schema.get_objectclasses()

# Get a list of ldap.schema.models.AttributeType objects for all attributeTypes supported by this instance
attributetypes = standalone.schema.get_attributetypes()

# Get a list of the server defined matching rules
matchingrules = standalone.schema.get_matchingrules()

# Get a single matching rule instance that matches the mr_name. Returns None if the matching rule doesn't exist
matchingrule = standalone.schema.query_matchingrule(matchingrule_name)

# Get a single ObjectClass instance that matches objectclassname. Returns None if the objectClass doesn't exist
objectclass = standalone.schema.query_objectclass(objectclass_name)
    
# Returns a tuple of the AttributeType, and what objectclasses may or must take this attributeType. Returns None if attributetype doesn't
(attributetype, may, must) = standalone.schema.query_attributetype(attributetype_name)

Offline utils

standalone.ldif2db(bename, suffixes, excludeSuffixes, encrypt, import_file)
standalone.db2ldif(bename, suffixes, excludeSuffixes, encrypt, repl_data, outputfile)
standalone.bak2db(archive_dir,bename=None)
standalone.db2bak(archive_dir)
standalone.db2index(bename=None, suffixes=None, attrs=None, vlvTag=None)
standalone.dbscan(bename=None, index=None, key=None, width=None, isRaw=False)
 
# Generate a simple ldif file using the dbgen.pl script, and set the ownership and permissions to match the user that the server runs as
standalone.buildLDIF(number_of_entries, path_to_ldif, suffix='dc=example,dc=com')

Contact us

If you have any issue or a question, you have a few options: