Grouper Provisioning: PSPNG (Legacy)
There is a new Grouper Provisioning Framework, released with Grouper 2.6 and above.
This page on PSPNG is no longer current as of 2022.
Provisioning's job is to reflect groups and their memberships in other systems. This is handled in Grouper via the PSPNG component, which is mostly a client of the Grouper changelog and is designed as a ChangeLog Consumer (CLC). Detected changes in Grouper via the changelog are picked by PSPNG and evaluated for provisioning operations selectively.
- 1 History
- 2 Development Status and Features
- 3 Configuration
- 4 Advanced Ldaptive Properties
- 5 Run PSPNG
- 6 Group/Folder Selection
- 7 Logging & Troubleshooting
- 8 Account creation
- 9 Full SYNC Provisioning
- 10 PSPNG Debugging
- 11 Provisioning "empty" groupOfNames groups
- 12 Provisioner Templates
- 12.1 Bushy DNs
- 12.2 GROUP OF UNIQUE NAMES
- 12.3 POSIX GROUPS
- 12.4 ACTIVE DIRECTORY GROUPS
- 12.5 USER ATTRIBUTES
History
Over the years, dozens of provisioners have been created -- some focused on a single destination type and others with some generic functionality combined with a very wide variety of options and capabilities.
Starting in 2014, the Grouper Team and Users concluded that the provisioning priorities should change: less flexibility and increased simplicity and performance. The Next Generation of the Grouper Provisioning Service Provider (aka PSPNG) was defined in Post PSP Provisioning .
As of Grouper 2.6 and above the PSPNG is no longer used. Please see Grouper Provisioning Framework documentation
Development Status and Features
PSPNG's general structure should be ready to provision various targets, but its current implementation is limited to provisioning LDAP targets like:
-(Unix) LDAP Groups: (GroupOfUniqueNames, GroupOfNames, PosixGroup)
-Active Directory Groups
-LDAP Attributes (like eduPersonEntitlement)
Configuration
PSPNG's Configuration is done via the conf/grouper-loader.properties file, in the grouper API Binary, with a paragraph for each provisioning destination, as well as an additional paragraph that enables and configures FullSync operation. There are several configuration options documented in the following spreadsheet:
Provisioner Type | Parameter | Default | Description | Default Behavior |
All Provisioners |
|
|
|
|
| enabled | true | Whether the provisioner will make changes. When this is false, the provisioner skips all changelog or full-sync requests. | The provisioner reflects changes in Grouper downstream to the target system. |
|
|
| Jexl expression that refers to stem_attributes, group_attributes, or group | Provision groups if <provisionerName> is in a group or stem provision_to attribute AND NOT in a do_not_provision_to attribute |
|
| etc:pspng:provision_to, etc:pspng:do_not_provision_to | List of Grouper Attributes used in group selection. Used to improve the speed of finding relevant groups. | Use the groupSelectionExpression against groups either with provision_to/do_not_provision_to Attributes or in folders with those attributes. |
|
| true | True: the values of the attributes listed in attributesUsedInGroupSelectionExpression are the provisioner names. False: The attributes listed in attributesUsedInGroupSelectionExpression have different values than the provisioner. | PSPNG does a database search for Attribute=<provisioner name> which narrows the number of groups to compare to the groupSelectionExpression. |
|
|
| How long should Grouper (Group, Stem, Subject) data be cached by the provisioners? | Grouper data will be cached for 10 minutes, though it is flushed when groups change. |
|
|
| How many Grouper Groups should be kept in memory at a time? |
|
|
|
| How many Grouper Subjects should be kept in memory at a time? |
|
|
|
| Does provisioniner need User/Subject information from the Target System? For example, do any JEXL expressions need information that is not available in Grouper Subjects? | All provisioning will be done based (only) on Grouper-Subject information. |
|
|
| Does provisioner need group information from the Target System? For example, this information could be used in various JEXL expressions. | All provisioning will need to be done based on Grouper-Group information |
|
|
| Only used when needsTargetSystemUsers=true: Should users be created when they cannot be found? | Users will not be created by Grouper Provisioning. Provisioning actions that require the users will fail. |
|
|
| Only used when needsTargetSystemUsers=true: How many users can be sought in a single Fetch? | Fetches will seek information for up to 50 users at a single time. |
|
|
| Only used when needsTargetSystemGroups=true: How many groups can be sought in a single Fetch? | Fetches will seek information for up to 50 groups at a time. |
|
|
| Can groups be created without any members? If so, it is easier to create them separately from membership changes. | Yes, create groups as soon as possible. |
|
|
| FullSync: Wait a bit before retrying a group that has failed. This prevents aggressive infinite loops. | 1 second pause before retrying a failed group. |
LdapProvisioner (Abstract) |
|
| What ldap pool should be used by this provisioner |
|
|
|
| Where to find users? | Required if provisioner needsTargetSystemUsers=true |
|
|
| Jexl expression that refers to stem_attributes, group_attributes, or group | How to find users in the Target System? |
|
|
| Comma-separated list of attributes that are useful for logging and that are needed by userSearchFilter or by a subclass's ValueFormats | Reads common attributes from either Unix or ActiveDirectory LDAP servers |
|
|
| Whether paging should be enabled in LDAP search requests; This typically requires the paging extension to be enabled and configured in LDAP to avoid LDAP Error Code 12. | TRUE |
|
|
| How many result objects can be pulled by a single request. This is small to avoid problems by default. | Break the results of a large query into fairly tiny chunks. |
|
|
| How long to keep User information in memory? | Keep User information in memory for 10 minutes, though user-information is flushed when users are changed by a provisioner |
|
|
| Deprecated: use the more general targetSystemUserCacheSize. | Keep the last 10000 users found by searching with Subject information |
|
|
| Is this an active-directory server? If so, then AD-specific attribute-value-paging is enabled. Also, member (reverse user-to-group virtual attribute) is enabled. | LDAP server is treated like a non-active-directory server. Problems will occur with full-sync of large groups. |
|
|
| How many values can be added/removed from an attribute in a single ldap operation | Breaks large list of values that need to be added/removed from an attribute into chunks that this size. For example, 5000 values that need to be added would be added in 50 chunks of 100 values each. |
| targetSystemUserCacheSize | 10000 |
|
|
|
| null | Warning: Grouper PSPNG is not a good provisioner for Accounts/Subjects. See 'Account Creation' section below. Where should account-/subject-objects be created when they don't already exist? This is only used if createMissingUsers=true | This is appended to the dn attribute produced by the userCreationLdifTemplate. |
| userCreationLdifTemplate | null | Warning: Grouper PSPNG is not a good provisioner for Accounts/Subjects. See 'Account Creation' section below. What account/subject ldap objects should be created when they don't already exist in the LDAP directory? This is only used if createMissingUsers=true |
|
LdapGroupProvisioner (Also Provisioner and LDAPProvisioner) |
|
| What attribute represents a group's members in the Target System? | Active Directory should just work. Otherwise, this is required. |
|
|
| What value (typically based on Subject or TargetSystemUser information) is written into the memberAttributeName attribute of groups? | Active Directory and GroupOfUniqueNames will typically work. This is a JEXL expression and is parsed at runtime. You may choose to script your way into this, by perhaps choosing a specific attribute:
|
|
|
| Virtual attribute of accounts that lists their groups |
|
|
|
| What LDIF should be written to the directory to add a group. Multiple lines need to be separated by || (double-pipes). The DN of the LDIF will be combined with groupCreationBaseDn> | For AD, limit to less than 1024 if sending description, like this: description: ${org.apache.commons.lang3.StringUtils.abbreviate(group.description == null ? null : group.description.replaceAll("\\r\\n"," ").replaceAll("\\n"," ").replaceAll("\\r"," "), 900)} |
|
|
| Where should groups be created? At group-creation time, this is appended to the DN that results from the groupCreationLdifTemplate. | Groups are created starting at the top of the search BaseDn. |
|
|
| Where are groups found? |
|
|
|
| If set to TRUE, groups under groupCreationBaseDn that are not in Grouper will be removed at the end of a full sync. | During full syncs, groups are not removed if they do not match the allGroupsSearchFilter or groupSelectionExpression. |
|
|
| FUTURE: How to find all the groups that grouper-provisioning maintains. If <grouperIsAuthoritative>, then groups found via this filter will be removed during a full sync. | Groups are not removed when they are removed from Grouper nor when they no longer match the groupSelectionExpression. |
|
|
| How to find a group, based on Grouper Group (or stem) information |
|
|
|
| Attributes that should be read from groups when searching for them. This needs to include all the attributes used in singleGroupSearchFilter. This should not include the attribute which holds the group's members. | Support common, basic singleGroupSearchFilters. |
|
|
| How long should LDAP-Group information be cached in memory? | Keep LDAP Group information in memory for 10 minutes, though it is flushed when users are changed by a provisioner. |
|
|
| How many LDAP groups to keep in memory, indexed by Grouper Group. |
|
|
|
| See above (JEXL expressions use User and Group information from the Target System) |
|
|
|
| See above (JEXL expressions use User and Group information from the Target System) |
|
| allowEmptyDnAttributeValues | FALSE | v2.5.51+ Set true to allow attributes that require DN syntax to be empty in order to support use of the null DN (also known as the zero-length DN). By doing so the PSP-NG is able to provision an "empty" group with objectClass groupOfNames and the attribute 'member' with no value (the null DN) | false, remove the attribute with no value from the LDIF before creating the group |
| removeNullDnFromGroupLdifCreationTemplate | FALSE | v2.5.51+ Set true so that LDIF from the group creation LDIF template is filtered and the membership attribute (e.g. member) with a null DN is removed before the LDIF is used to compare an existing group in the LDAP directory with the computed LDIF. | false, no filtering of the group creation LDIF is done to remove a membership attribute with a null DN. |
LdapAttributeProvisioner |
|
| What attribute is changed in User LDAP objects to represent group membership? |
|
|
|
| What value (typically based on Subject or TargetSystemUser information) is written into the provisionedAttributeName users? | The stem:Group name is written to the attribute specified in <provisionedAttributeName> |
|
|
| See above (JEXL expressions only use User information from the Target System) |
|
|
|
| See above (JEXL expressions only use User information from the Target System) |
|
|
|
| What values of the attribute is grouper authoritative for during a full sync? null (default) or empty means that pspng will only process removals as memberships change, and won't clean up unknown attribute values. | Warning: Grouper should have full control over the target attribute to avoid complications that come from sharing attributes with multiple provisioning tools. |
PSPNG JEXL object notations
JEXL Syntax | Description | Example string |
|---|---|---|
| This will return the full grouper "name" for the group that is part of the event. | folderA:folderB:FolderC:GroupDisplayName |
| This will return an attribute from the LDAP user ( that is likely found via a search during the event ) | Depends on the data in the connected Ldap service. |
LDAP Properties
LDAP configuration is done based on the ldaptive library's property configuration. A paragraph of ldap configuration is created in grouper-loader.properties for each LDAP endpoint, and that paragraph is referenced by the appropriate provisioners. Comments in grouper-loader.base.properties refers to these "LDAP pools" as "LDAP connections".
Targeting Active Directory
Where Active Directory is the target environment, make sure you are pointing to a FQDN with active/standby load balancing or to a primary node. Other forms of load balancing can lead to inconsistent results or AD conflict CNF objects.
At this time, the LDAP bindCredential cannot be encrypted via the Grouper morphstring.
|
Advanced Ldaptive Properties
PSPNG relies on the Ldaptive library for all LDAP-related operations. To learn more about what other LDAP properties are available, one simple example can be found here. Moving into more realistic examples will probably be helped by looking at the ldaptive configuration classes and the setters available within them: connections, pooling, binding (sasl, gssapi, x509, jks, etc).
In case it is helpful, this is currently implemented here. You may also wish to take a look at GRP-1306 to learn more about the differences between vtdap (the previously used LDAP library) vs ldaptive.
Run PSPNG
PSPNG runs as part of the grouper loader. So simply run the loader with:
cd <grouper-api-binary-folder>/bin
./gsh -loaderGroup/Folder Selection
Groups can be enabled and disabled based on Group or Folder/Stem information and attributes. By default, two multivalued attributes are used:
Attribute Definition | Attribute Name | Description |
|---|---|---|
etc:pspng:provision_to_def | etc:pspng:provision_to | Enable downstream provisioning |
etc:pspng:do_not_provision_to_def | etc:pspng:do_not_provision_to | Disable downstream provisioning |
The groupSelectionExpression can be modified to look at different group characteristics or group/folder attributes. If you replace the default provision_to/do_not_provision_to expression with an expression that does not reference Grouper Attributes, you'll need to still put provision_to on the parent folder(s) of the groups you wish to provision. GRP-1903 will fix this so you'll be able to set attributesUsedInGroupSelectionExpression to an empty value. If you use an expression that uses other Attributes, then make sure attributesUsedInGroupSelectionExpression is a comma-separated list of the attributes you need.
How do I create PSPNG attribute definitions?
These attribute definitions are auto-created by Grouper the very first time PSPNG runs. If you have configured PSPNG to run on startup, then the attributes will be created then or else you may need to wait for the scheduled PSPNG job to kick in.
If you need to, you can always create these via the Grouper Shell.
Comments
This question has "bothered me" for a while too. But I never really went to look at it.
Here are my 2 cents.
Docs say: "LDAP configuration is done based on the ldaptive library's property configuration."
CODE REF: .../src/grouper/edu/internet2/middleware/grouper/ldap/ldaptive/LdaptiveSessionImpl.java
This some project (psudo) code with some extra comments from me.
"
private static Properties getLdaptiveProperties(String ldapSystemName) {
Properties _ldaptiveProperties = new Properties();
String ldapPropertyPrefix = "ldap." + ldapSystemName + ".";
....
// walk all the grouper-loader.properties file
for (String propName : GrouperLoaderConfig.retrieveConfig().propertyNames()) {
/* if the property from the file starts with "ldap." + ldapSystemName + "." */
if ( propName.startsWith(ldapPropertyPrefix) ) {
String propValue = GrouperLoaderConfig.retrieveConfig().propertyValueString(propName, "");
/* Get the part of the property after ldapPropertyPrefix ( "ldap." + ldapSystemName + "." ) */
String propNameTail = propName.substring(ldapPropertyPrefix.length());
/* avoid NPE on values */
if (propValue == null) {
propValue = "";
}
/*
* assume all configured values from "ldap.*.<propNameTail>" align with the "org.ldaptive."+propNameTail attributes
* /
_ldaptiveProperties.put("org.ldaptive." + propNameTail, propValue);
/*
* then there are some compatibility between old vtldap properties and ldaptive versions if blocks
*/
// Some compatibility between old vtldap properties and ldaptive versions
// url (vtldap) ==> ldapUrl
...
// tls (vtldap) ==> useStartTls
...
// user (vtldap) ==> bindDn
...
// pass (vtldap) ==> bindCredential
...
// countLimit (vtldap) ==> sizeLimit
...
// timeout (vtldap) ==> connectTimeout
...
}
}
"
I would suggest *not* using the "vtldap" terms as they are only there for backwards compatibility.
AKA: (url, tls, user, pass, countLimit, timeout ) are only for "backwards compatibility".
Please use ( ldapUrl, useStartTls, bindDn, bindCredential, sizeLimit, connectTimeout ) or any other "org.ldaptive.*" property would be supported too.
Hope that helps.
Looks like that is right for PSPNG, but the UI is still looking for the old values. If you use .ldapUrl, for instance, you can't create an LDAP loader job from the UI. It's looking for ".url" not ".ldapUrl" so the paragraph won't be visible. Note the pattern on 286 in GrouperLoaderContainer.java. Worth creating a JIRA issue for this?
Relatively simple fix for this issue, but I don't know where else something outside of PSPNG is consuming that property. It'd have to be changed there too to check for both values.
/**
* ldap server ids
* @return ldap server ids
*/
public List<ConfigName> getLdapServerIds() {
List<ConfigName> result = new ArrayList<ConfigName>();
GrouperLoaderConfig grouperLoaderConfig = GrouperLoaderConfig.retrieveConfig();
Pattern pattern = Pattern.compile("^ldap.([^.]+).url$");
for (String propertyName : grouperLoaderConfig.propertyNames()) {
Matcher matcher = pattern.matcher(propertyName);
if (!matcher.matches()) {
continue;
}
String configUrlName = matcher.group(1);
String configUrl = grouperLoaderConfig.propertyValueString(propertyName);
result.add(new ConfigName(configUrlName, configUrlName + " - " + configUrl));
}
return result;
}created :
Looking at grouperIsAuthoritative:
Groups are not removed when they are removed from Grouper nor when they no longer match the groupSelectionExpression.
This is only the case for full-syncs, correct? It's been my experience that the change log consumer will still try to delete a group if it exists in the target system, even if grouperIsAuthoritative is false.
That is correct. I also neglected to mention that it won't be deleted during Full Syncs if the group falls out of the groupSelectionExpression as well. Thanks for the catch.
Above the ldap paragraphs say the following:
ldap.groupOfNames.ldapUrl = ldaps: //hostname ldap.groupOfNames.bindDn = cn=xxxxxx,ou=xxxxx ldap.groupOfNames.bindCredential = xxxxx ldap.groupOfNames.someOtherLdapProperty = value# in another paragraph a provisioner is associated with the LDAP pool namechangeLog.consumer.pspng_groupOfUniqueNames.ldapPoolName = groupOfNamesLooking at grouper-loader.base.properties, the bindDn and password should be specified like this:
ldap.groupOfNames.ldapUrl = ldaps: //hostname ldap.groupOfNames.user= cn=xxxxxx,ou=xxxxx ldap.groupOfNames.pass = xxxxx ldap.groupOfNames.someOtherLdapProperty = value# in another paragraph a provisioner is associated with the LDAP pool namechangeLog.consumer.pspng_groupOfUniqueNames.ldapPoolName = groupOfNames Does this make a difference?