Skip to end of metadata
Go to start of metadata

Provisioning's job is to reflect Groups and their Memberships in other systems. This is handled in Grouper via the PSPNG component, which is mostly a client of the Grouper changelog and is designed as a ChangeLog Consumer (CLC). Detected changes in Grouper via the changelog are picked by PSPNG and evaluated for provisioning operations selectively.

 

History

Over the years, dozens of provisioners have been created -- some focused on a single destination type and others with some generic functionality combined with a very wide variety of options and capabilities.

In recent years, the Grouper Team and Users have concluded that the provisioning priorities should change: less flexibility and increased simplicity and performance. The Next Generation of the Grouper Provisioning Service Provider (aka PSPNG) was defined in Post PSP Provisioning and is expected to replace the older PSP components over time.

Development Status and Features

PSPNG's general structure should be ready to provision various targets, but its current implementation is limited to provisioning LDAP targets like:

  • -(Unix) LDAP Groups: (GroupOfUniqueNames, GroupOfNames, PosixGroup)
  • -Active Directory Groups
  • -LDAP Attributes (like eduPersonEntitlement)

Configuration

PSPNG's Configuration is done via the conf/grouper-loader.properties file, in the grouper API Binary, with a paragraph for each provisioning destination, as well as an additional paragraph that enables and configures FullSync operation. There are several configuration options documented in the following spreadsheet:

 

Provisioner TypeParameterDefaultDescriptionDefault Behavior
All Provisioners
provisionerName
<required>
  
 
groupSelectionExpression
${
  utils.containedWithin(provisionerName,
stemAttributes['etc:pspng:provision_to'],
groupAttributes['etc:pspng:provision_to'])
  &&  
!utils.containedWithin(provisionerName,
stemAttributes['etc:pspng:do_not_provision_to'],
groupAttributes['etc:pspng:do_not_provision_to'])
}
Jexl expression that refers to stem_attributes, group_attributes, or groupProvision groups if <provisionerName> is in a group or stem provision_to attribute AND NOT in a do_not_provision_to attribute
 
grouperDataCacheTime_secs
600 (seconds)
How long should Grouper (Group, Stem, Subject) data be cached by the provisioners?Grouper data will be cached for 10 minutes, though it is flushed when groups change.
 
grouperGroupCacheSize
10000 (Groups)
How many Grouper Groups should be kept in memory at a time? 
 
grouperSubjectCacheSize
10000 (Subjects)
How many Grouper Subjects should be kept in memory at a time? 
 
needsTargetSystemUsers
FALSE
Does provisioniner need User/Subject information from the Target System? For example, do any JEXL expressions need information that is not available in Grouper Subjects?All provisioning will be done based (only) on Grouper-Subject information.
 
needsTargetSystemGroups
FALSE
Does provisioner need group information from the Target System? For example, this information could be used in various JEXL expressions.All provisioning will need to be done based on Grouper-Group information
 
createMissingUsers
FALSE
Only used when needsTargetSystemUsers=true: Should users be created when they cannot be found?Users will not be created by Grouper Provisioning. Provisioning actions that require the users will fail.
 
userSearch_batchSize
50
Only used when needsTargetSystemUsers=true: How many users can be sought in a single Fetch?Fetches will seek information for up to 50 users at a single time.
 
groupSearch_batchSize
50
Only used when needsTargetSystemGroups=true: How many groups can be sought in a single Fetch?Fetches will seek information for up to 50 groups at a time.
 
supportsEmptyGroups
TRUE
Can groups be created without any members? If so, it is easier to create them separately from membership changes.Yes, create groups as soon as possible.
 
sleepTimeAfterError_ms
1000
FullSync: Wait a bit before retrying a group that has failed. This prevents aggressive infinite loops.1 second pause before retrying a failed group.
LdapProvisioner (Abstract)
ldapPoolName
<required>
What ldap pool should be used by this provisioner 
 
userSearchBaseDn
null
Where to find users?Required if provisioner needsTargetSystemUsers=true
 
userSearchFilter
null
Jexl expression that refers to stem_attributes, group_attributes, or groupHow to find users in the Target System?
 
userSearchAttributes[]
dn,cn,uid,mail,
samAccountName,uidNumber,
objectclass
Comma-separated list of attributes that are useful for logging and that are needed by userSearchFilter or by a subclass's ValueFormatsReads common attributes from either Unix or ActiveDirectory LDAP servers
 
searchResultPagingEnabled
TRUE
Whether paging should be enabled in LDAP search requests; This typically requires the paging extension to be enabled and configured in LDAP to avoid LDAP Error Code 12.TRUE
 
ldapSearchResultPagingSize
100
How many result objects can be pulled by a single request. This is small to avoid problems by default.Break the results of a large query into fairly tiny chunks.
 
ldapUserCacheTime_secs
600
How long to keep User information in memory?Keep User information in memory for 10 minutes, though user-information is flushed when users are changed by a provisioner
 
ldapUserCacheSize
10000
How many LDAP accounts can be kept in memory at a time, indexed by the Subject mapped to them?Keep the last 10000 users found by searching with Subject information
 
isActiveDirectory
FALSE
Is this an active-directory server? If so, then AD-specific attribute-value-paging is enabled. Also, member (reverse user-to-group virtual attribute) is enabled.LDAP server is treated like a non-active-directory server. Problems will occur with full-sync of large groups.
 
maxValuesToChangePerOperation
100
How many values can be added/removed from an attribute in a single ldap operationBreaks large list of values that need to be added/removed from an attribute into chunks that this size. For example, 5000 values that need to be added would be added in 50 chunks of 100 values each.
LdapGroupProvisioner (Also Provisioner and LDAPProvisioner)
memberAttributeName
'member' for AD
<required> otherwise
What attribute represents a group's members in the Target System?Active Directory should just work. Otherwise, this is required.
 
memberAttributeValueFormat
${ldapUser.dn}
What value (typically based on Subject or TargetSystemUser information) is written into the memberAttributeName attribute of groups?

Active Directory and GroupOfUniqueNames will typically work. This is a JEXL expression and is parsed at runtime. You may choose to script your way into this, by perhaps choosing a specific attribute:

${ldapUser.getStringValue("uid")}
 
groupAttributeName
memberof for AD
null otherwise
Virtual attribute of accounts that lists their groups 
 
groupCreationLdifTemplate
null
What LDIF should be written to the directory to add a group. Multiple lines need to be separated by || (double-pipes). The DN of the LDIF will be combined with groupCreationBaseDn> 
 
groupCreationBaseDn
<groupSearchBaseDn>
Where should groups be created? At group-creation time, this is appended to the DN that results from the groupCreationLdifTemplate.Groups are created starting at the top of the search BaseDn.
 
groupSearchBaseDn
<required>
Where are groups found? 
 
grouperIsAuthoritative
FALSE
Should groups in the groupSearchBaseDn/allGroupSearchFilter be removed if they no longer exist in Grouper? 
 
allGroupSearchFilter
null
FUTURE: How to find all the groups that grouper-provisioning maintains. If <grouperIsAuthoritative>, then groups found via this filter will be removed during a full sync.Groups are not removed when they are removed from Grouper nor when they no longer match the groupSelectionExpression.
 
singleGroupSearchFilter
<required>
How to find a group, based on Grouper Group (or stem) information 
 
groupSearchAttributes
cn,gidNumber,samAccountName,objectclass
Attributes that should be read from groups when searching for them. This needs to include all the attributes used in singleGroupSearchFilter. This should not include the attribute which holds the group's members.Support common, basic singleGroupSearchFilters.
 
ldapGroupCacheTime_secs
600
How long should LDAP-Group information be cached in memory?Keep LDAP Group information in memory for 10 minutes, though it is flushed when users are changed by a provisioner.
 
ldapGroupCacheSize
10000
How many LDAP groups to keep in memory, indexed by Grouper Group. 
 
needsTargetSystemUsers
TRUE
See above (JEXL expressions use User and Group information from the Target System) 
 
needsTargetSystemGroups
TRUE
See above (JEXL expressions use User and Group information from the Target System) 
LdapAttributeProvisioner
(Also Provisioner and LDAPProvisioner)
provisionedAttributeName
<required>
What attribute is changed in User LDAP objects to represent group membership? 
 
provisionedAttributeValueFormat
${group.name}
What value (typically based on Subject or TargetSystemUser information) is written into the provisionedAttributeName users?The stem:Group name is written to the attribute specified in <provisionedAttributeName>
 
needsTargetSystemUsers
TRUE
See above (JEXL expressions only use User information from the Target System) 
 
needsTargetSystemGroups
FALSE
See above (JEXL expressions only use User information from the Target System) 
 
allProvisionedAttributePrefix
null
What values of the attribute is grouper authoritative for during a full sync? null (default) or empty means that pspng will only process removals as memberships change, and won't clean up unknown attribute values.Warning: Grouper should have full control over the target attribute to avoid complications that come from sharing attributes with multiple provisioning tools.

LDAP Properties

 

LDAP configuration is done based on the ldaptive library's property configuration. A paragraph of ldap configuration is created in grouper-loader.properties for each LDAP endpoint, and that paragraph is referenced by the appropriate provisioners. 

 

At this time, the LDAP bindCredential cannot be encrypted via the Grouper morphstring.

ldap.groupOfNames.ldapUrl = ldaps://hostname
ldap.groupOfNames.bindDn = cn=xxxxxx,ou=xxxxx
ldap.groupOfNames.bindCredential = xxxxx
ldap.groupOfNames.someOtherLdapProperty = value
  
# The LDAP block is connected to each provisioner via the LDAP pool name
changeLog.consumer.pspng_groupOfUniqueNames.ldapPoolName = groupOfNames

Advanced Ldaptive Properties

PSPNG relies on the Ldaptive library for all LDAP-related operations. To learn more about what other LDAP properties are available,  one simple example can be found here. Moving into more realistic examples will probably be helped by looking at the ldaptive configuration classes and the setters available within them: connectionspooling, binding (sasl, gssapix509, jks, etc).

In case it is helpful, this is currently implemented here. You may also wish to take a look at GRP-1306 to learn more about the differences between vtdap (the previously used LDAP library) vs ldaptive.

Run PSPNG

PSPNG runs as part of the grouper loader. So simply run the loader with:

cd <grouper-api-binary-folder>/bin
./gsh -loader

 

Group/Folder Selection

Groups can be enabled and disabled based on Group or Folder/Stem information and attributes. By default, two multivalued attributes are used:

Attribute DefinitionAttribute NameDescription

etc:pspng:provision_to_def

etc:pspng:provision_toEnable downstream provisioning

etc:pspng:do_not_provision_to_def

etc:pspng:do_not_provision_toDisable downstream provisioning


How do I create PSPNG attribute definitions?

These attribute definitions are auto-created by Grouper the very first time PSPNG runs. If you have configured PSPNG to run on startup, then the attributes will be created then or else you may need to wait for the scheduled PSPNG job to kick in.

If you need to, you can always create these via the Grouper Shell. 

How do I assign PSPNG attribute definitions?

These attributes need to be assigned to Groups or Folders via the Lite UI. 

 

After the attribute is assigned to a folder or group, you need to assign it a value. The value of the attribute MUST match the provisioner name that is defined in the grouper-loader.properties configuration (i.e. pspng_groupOfUniqueNames).

PSPNG will evaluate whether a group or stem/folder qualifies for provisioning by running the group-selecting filter as a JEXL expression. The expression is able to process and evaluate attribute definitions on a given group/folder and returns true/false to indicate whether PSPNG should continue with downstream provisioning.

Logging & Troubleshooting

To troubleshoot PSPNG configuration, you may need to add the following logging configuration to the log4j.properties file:

 

log4j.logger.edu.internet2.middleware.grouper.pspng=DEBUG
log4j.logger.edu.internet2.middleware.grouper.changeLog=DEBUG


Account creation

To facilitate (load) testing, Grouper PSPNG has limited ability to create accounts in a target system. This ability is very rudimentary and should not be used to maintain accounts in production; such maintenance should be done by an IdM product or some other ETL mechanism. If you choose to ignore this best practice, you'll probably run into two main problems:

  1. Grouper/PSPNG has no ability to keep accounts up to date; it only creates them if they're missing, and
  2. Your Grouper-subject mappings probably don't have some of the information needed for account provisioning. And, even if all you need is name, username, and email address (which are probably in your subject mappings), you'll still run into problem (1) where Grouper offers no mechanism to update name and email address when they change in your subject source.

 

Full SYNC Provisioning

 

Upgrade warning (PSPNG Patch #11, June 8, 2017)

The Full-Sync-provisioning improvements in Grouper 2.3/PSPNG Patch #11 are configured with new, better properties. (You should no longer use the properties that start with changeLog.psp.fullSync.) 

 

In the background of PSPNG, there is always a full-sync-provisioning engine running which is automatically used when incremental provisioning finds conflicting changes or otherwise is unable to handle the changelog events. The full-sync items in grouper-loader.properties do not alter/configure the background engine but instead define quartz jobs that send all the groups marked for provisioning into the queues that drive the full-sync engine. Additionally, if a provisioner is "authoritative" (changeLog.consumer.<provisioner>.grouperIsAuthoritative=true), then a cleanup task is included in the scheduled job which will delete extra groups or attributes that no longer exist in the Group Registry. (Note: Data in a provisioning target will always get deleted by incremental provisioning when groups are deleted, regardless whether a provisioner is authoritative. Full-sync cleanup is just a safety net in case incremental provisioning misses something, or if the data was written into the target system outside of PSPNG.)

Each provisioner that wants to schedule periodic full-syncs will need the following lines included in grouper-loader.properties:

Full Sync Provisioner
otherJob.<provisioner-name>_full.class = edu.internet2.middleware.grouper.pspng.FullSyncStarter
otherJob.<provisioner-name>_full.quartzCron = 0 0 0 * * ?   #Every midnight

# Note, there is presently no "runAtStartup" option as there was before PSPNG Patch #11. Please see GRP-1563.

<provisioner-name> is the 3rd field of the properties used to configure the rest of the provisioner. For instance, <provisioner-name> is xyz for the following config snippet:

changeLog.consumer.xyz.class = edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
changeLog.consumer.xyz.type = edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner

Provisioner Templates

The syntax for each line is:

changelog.consumer.<provisioner-name>.<propertyName> = value

Bushy DNs

The DN of the group can be Bushy (one LDAP OU for each Folder/Stem of the group) by doing the following:

dn: cn=${group.name},${utils.bushyDn(group.name, "cn", "ou")},dc=example,dc=edu


GROUP OF UNIQUE NAMES

This is most likely the provisioner needed for most LDAP servers, such as OpenLDAP, OpenDJ, etc.
changeLog.consumer.pspng_groupOfUniqueNames.class = edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
changeLog.consumer.pspng_groupOfUniqueNames.type = edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
changeLog.consumer.pspng_groupOfUniqueNames.quartzCron = 0 * * * * ?
changeLog.consumer.pspng_groupOfUniqueNames.ldapPoolName = opendj
changeLog.consumer.pspng_groupOfUniqueNames.memberAttributeName = uniqueMember
# changeLog.consumer.pspng_posixGroup.memberAttributeValueFormat = ${ldapUser.getStringValue("uid")}
changeLog.consumer.pspng_groupOfUniqueNames.memberAttributeValueFormat = ${ldapUser.getDn()}
changeLog.consumer.pspng_groupOfUniqueNames.groupSearchBaseDn = ou=grouper,ou=groups,dc=example,dc=edu
changeLog.consumer.pspng_groupOfUniqueNames.allGroupsSearchFilter = objectclass=groupOfUniqueNames
changeLog.consumer.pspng_groupOfUniqueNames.singleGroupSearchFilter = (&(objectclass=groupOfUniqueNames)(cn=${group.name}))
changeLog.consumer.pspng_groupOfUniqueNames.groupSearchAttributes=cn,gidNumber,objectclass
changeLog.consumer.pspng_groupOfUniqueNames.groupCreationLdifTemplate = dn: cn=${group.name}||cn: ${group.name}||objectclass: groupOfUniqueNames
changeLog.consumer.pspng_groupOfUniqueNames.userSearchBaseDn = cn=users,dc=example,dc=edu
changeLog.consumer.pspng_groupOfUniqueNames.userSearchFilter = uid=${subject.id}

 

POSIX GROUPS

changeLog.consumer.pspng_posixGroup.class = edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
changeLog.consumer.pspng_posixGroup.type = edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
changeLog.consumer.pspng_posixGroup.quartzCron = 0 * * * * ?
changeLog.consumer.pspng_posixGroup.ldapPoolName = opendj
changeLog.consumer.pspng_posixGroup.memberAttributeName = memberUid
changeLog.consumer.pspng_posixGroup.memberAttributeValueFormat = ${ldapUser.getStringValue("uid")}
changeLog.consumer.pspng_posixGroup.groupSearchBaseDn = ou=grouper-posix,ou=groups,dc=example,dc=edu
changeLog.consumer.pspng_posixGroup.allGroupsSearchFilter = objectclass=posixGroup
changeLog.consumer.pspng_posixGroup.singleGroupSearchFilter = (&(objectclass=posixGrouper)(cn=${group.name}))
changeLog.consumer.pspng_posixGroup.groupSearchAttributes=cn,gidNumber,objectclass
# Obviously, gidNumber should be based on a grouper-group attribute
changeLog.consumer.psong_posixGroup.groupCreationLdifTemplate = dn: cn=posix-${group.name}||cn: posix-${group.name}||objectclass: posixGroup||objectclass: groupOfNames||gidNumber: ${group.idIndex}
changeLog.consumer.pspng_posixGroup.userSearchBaseDn = cn=users,dc=example,dc=edu
changeLog.consumer.pspng_posixGroup.userSearchFilter = uid=${subject.id}

 

ACTIVE DIRECTORY GROUPS

changeLog.consumer.pspng_activedirectory.class = edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
changeLog.consumer.pspng_activedirectory.type = edu.internet2.middleware.grouper.pspng.LdapGroupProvisioner
changeLog.consumer.pspng_activedirectory.quartzCron = 0 * * * * ?
changeLog.consumer.pspng_activedirectory.ldapPoolName = active_directory
changeLog.consumer.pspng_activedirectory.isActiveDirectory = true
changeLog.consumer.pspng_activedirectory.memberAttributeName = member
changeLog.consumer.pspng_activedirectory.memberAttributeValueFormat = ${ldapUser.getDn()}
changeLog.consumer.pspng_activedirectory.groupSearchBaseDn = ou=grouper,ou=groups,dc=example,dc=edu
changeLog.consumer.pspng_activedirectory.allGroupsSearchFilter = objectclass=group
changeLog.consumer.pspng_activedirectory.singleGroupSearchFilter = (&(objectclass=group)(cn=${group.name}))
changeLog.consumer.pspng_activedirectory.groupCreationLdifTemplate = dn: cn=${group.name}||cn: ${group.name}||objectclass: group
changeLog.consumer.pspng_activedirectory.userSearchBaseDn = cn=users,dc=example,dc=edu
changeLog.consumer.pspng_activedirectory.userSearchFilter = samAccountName=${subject.id}


USER ATTRIBUTES

changeLog.consumer.pspng_attributes.class = edu.internet2.middleware.grouper.pspng.PspChangelogConsumerShim
changeLog.consumer.pspng_attributes.type = edu.internet2.middleware.grouper.pspng.LdapAttributeProvisioner
changeLog.consumer.pspng_attributes.quartzCron = 0 * * * * ?
changeLog.consumer.pspng_attributes.retryOnError = true
changeLog.consumer.pspng_attributes.ldapPoolName = opendj
changeLog.consumer.pspng_attributes.provisionedAttributeName = eduPersonEntitlement
changeLog.consumer.pspng_attributes.provisionedAttributeValueFormat = g:${group.name}
changeLog.consumer.pspng_attributes.userSearchBaseDn = cn=users,dc=example,dc=edu
changeLog.consumer.pspng_attributes.userSearchFilter = uid=${subject.id}
changeLog.consumer.pspng_attributes.allProvisionedValuesPrefix=g:
  • No labels