Grouper Messaging provisioner

The Messaging provisioner publishes a message to a queue or topic every time a group, an entity (user), or a membership is added, updated, or removed in the scope of the provisioner. Downstream applications consume those messages to keep their own data in step with Grouper.

It is a membership-objects provisioner that sends to a message broker instead of writing to a directory, database, or SaaS API.

• Provisioner class: edu.internet2.middleware.grouper.app.messagingProvisioning.GrouperMessagingProvisioner

• Provisioning type: membershipObjects

• Supported brokers: Grouper built-in messaging, ActiveMQ, Amazon AWS SQS, RabbitMQ

• Message format: EsbEventJson, or a JEXL TranslationScript

See the Grouper messaging system page for the broker-specific setup (external system configuration, credentials, queue naming).

Most provisioners (LDAP, SQL, Entra ID, Google, and so on) keep a target system in step with Grouper. They can read the target, compare it to what Grouper expects, and correct differences. The Messaging provisioner cannot do that, because a broker is a one-way pipe: there is nothing to read back. That changes how it behaves in several ways.

The consequences to plan for:

• Consumers must be idempotent. Because the provisioner sends events and does not confirm that they were applied, a consumer should handle a repeated message safely, using the object id and sequenceNumber.

• A new consumer that needs existing data needs a bootstrap. Messages are sent once. A consumer that starts listening later, or a queue that was purged, does not receive the earlier events. Load the initial state some other way (for example a web service or SQL export) and then apply messages from that point. See Grouper Messaging System - full sync messages for the design discussion.

• Scope selection is the main tool. As with every provisioner, use provisioning metadata on folders and groups to decide what is sent. Only in-scope objects produce messages.

Older versions of Grouper sent messages with an ESB change log consumer (EsbConsumer with EsbMessagingPublisher). The Messaging provisioner is the recommended replacement. It uses the same JSON envelope (esbEvent), so existing consumers generally keep working, and it adds:

• Granular control over which folders and groups are eligible, using the provisioning framework metadata

• Provisioner sync tracking and diagnostics in the UI

• Message content derived from what the provisioner computed, rather than raw change log entries

Configure it through the provisioner wizard in the UI (choose the Messaging "start with"), or in grouper-loader.properties / the database configuration. Properties are provisioner.<configId>.<property>.

The queue or topic is created automatically if it does not exist when loader.messaging.settings.autocreate.objects is true (the default).

Every message is a JSON envelope with a single event in the esbEvent array.

Empty values are left out of the message, which is why the examples below do not show description or parentStemId.

Entity messages use the event types MEMBER_* because they follow the ESB change log naming, where a Grouper "member" is what the provisioning framework calls an entity. A *_DELETE message has the same body as the corresponding *_ADD with a different eventType.

Sent when a group comes into scope, changes, or leaves scope.

Sent when a user comes into scope, changes, or leaves scope.

subjectSourceId and subjectIdentifier0 are taken from the entity attributes of the provisioner, so they appear when the entity data is available to the provisioner (from the subject source or the entity resolver).

Sent when a user is added to or removed from a group in scope. The membership is always reported as flattened, so an indirect member (through a subgroup) produces the same message as a direct member.

The three example messages fit together: they add group test:testGroup, then user banderson, then the membership between them, with sequenceNumber 2, 3 and 4. A consumer can join the messages using groupId and memberId.

If the consumer needs a different body than the standard one, set messagingFormatType to TranslationScript and give a JEXL script for the object types you want to customize. Each script returns a map, and that map becomes the event object in the esbEvent array. An object type with no script keeps the standard EsbEventJson body.

eventType is the event being sent (for example MEMBER_ADD or MEMBER_DELETE), so one script can return a smaller body for deletes. It is read only: the framework still sets the eventType field of the message itself.

Example entity script:

The script is JEXL, not JSON: quoted keys look like JSON but a trailing comma after the last entry, e.g. ..."email": targetEntity.email,\n }, is a parse error, not a warning. Drop the comma after the last entry.

As with the standard body, keys whose value is null (for example an entity with no email) are left out of the message, and a null property does not cause an error. Because of that, a misspelled variable name is also treated as null rather than reported. The envelope and the framework fields eventType, sequenceNumber, changeOccurred and createdOnMicros are always added after the script runs, so a script cannot override them. If the script does not return a map, or fails, the change is not sent and the provisioning run reports the error.

The framework's own createdOnMicros is not available to the script (it is computed after the script runs), but a script can add its own current-time value with a JEXL expression. JEXL2 has no new operator, so new java.util.Date() or new SimpleDateFormat(...) will not parse; java.time's static factory methods work instead, for example:

For the full list of properties and methods available on targetGroup, targetEntity and targetMembership, and the eventType values, see JEXL translation script reference at the end of this page.

Both run the same message-building code, so the messages are identical. What differs is how the provisioner decides what to send. Because the broker cannot be read, both compare against the provisioning sync tables, where Grouper records for each group, entity and membership whether it has been sent (in target).

A few consequences:

• A first full sync is a bootstrap. On a new provisioner nothing is recorded as sent, so a full sync sends an add message for every in-scope group, entity and membership. A large scope produces a large burst of messages, so size the queue and consumer accordingly.

• Full sync will not repair a consumer. If a message reached the broker and the consumer lost it, Grouper's record says it was sent, and a later full sync sends nothing. Recovering a consumer means replaying the data (a bootstrap) or clearing the provisioner's sync state so that everything is sent again.

• Incremental can escalate to a full sync. When a run has a large number of events (past the scoreConvertToFullSyncThreshold for the provisioner) the incremental job hands over to a full sync in the same run rather than process each event.

• Full sync updates are rare. Nothing is read back from the broker, so there are no attribute comparisons against the target to produce updates. Expect adds and deletes.

The provisioner takes the next sequenceNumber from its sync job (one counter per provisioner config id), and increments it for every object it sends. Messages are published in the order the provisioner processes them, and sequenceNumber reflects that order. Because the counter belongs to one provisioner, do not compare sequence numbers between different Messaging provisioners.

A gap in the sequence numbers can indicate a message that was lost or that has not been delivered yet. The provisioner does not resend a message that the broker accepted and later lost.

Properties and methods available to the groupFormatTranslationScript, entityFormatTranslationScript and membershipFormatTranslationScript scripts described under Custom message body with JEXL above.

The same event types as the standard body (see the Message format section above):

targetGroup, targetEntity and targetMembership are Java beans, so a JEXL script can use either the dotted property form (targetEntity.email) or the getter method directly (targetEntity.getEmail()). The ticket example that introduced this feature mixes both styles.

These methods are common to all three objects. What attribute names they accept depends on what the provisioner is configured to load (for example, an entity attribute only resolves if the provisioner's entity attribute resolver is configured to fetch it):

targetGroup (a ProvisioningGroup) built-in properties:

targetEntity (a ProvisioningEntity) built-in properties:

targetMembership (a ProvisioningMembership) built-in properties:

targetMembership does not carry fieldId, fieldName, membershipType or sourceId the way the standard membership body does. Those are filled in by the messaging layer, not stored on the object. Use targetGroup/targetEntity attributes, or id, to identify the membership; use targetGroup.retrieveAttributeValueString(...) / targetEntity.retrieveAttributeValueString(...) for anything else the script needs to look up.

targetEntity and targetGroup also each expose a provisioningEntityWrapper / provisioningGroupWrapper property, which reaches a few provisioning-framework lookups that are not on the target object itself (targetMembership.provisioningEntityWrapper and targetMembership.provisioningGroupWrapper reach the same wrappers as the entity and group scripts get directly). The wrapper classes have other methods too, mostly internal bookkeeping (matching Grouper-side and target-side copies, sync table rows, error state); these are the ones useful from a script:

Example, adding whether the entity is in an "owners" group to the entity message:

• Messages contain user identifiers (subjectId, subjectIdentifier0). Restrict access to the queue or topic accordingly.

• A change can be sent again if a run fails partway and is retried, so consumers should tolerate duplicates.

• The Grouper messaging system page lists the receive-side options, including the messaging listener daemon and the built-in grouper_message table cleanup.