Grouper Messaging provisioner

The Messaging provisioner publishes a message to a queue or topic every time a group, an entity (user), or a membership is added, updated, or removed in the scope of the provisioner. Downstream applications consume those messages to keep their own data in step with Grouper.

It is a membership-objects provisioner that sends to a message broker instead of writing to a directory, database, or SaaS API.

• Provisioner class: edu.internet2.middleware.grouper.app.messagingProvisioning.GrouperMessagingProvisioner

• Provisioning type: membershipObjects

• Supported brokers: Grouper built-in messaging, ActiveMQ, Amazon AWS SQS, RabbitMQ

• Message format: EsbEventJson (the only format today)

See the Grouper messaging system page for the broker-specific setup (external system configuration, credentials, queue naming).

Most provisioners (LDAP, SQL, Entra ID, Google, and so on) keep a target system in step with Grouper. They can read the target, compare it to what Grouper expects, and correct differences. The Messaging provisioner cannot do that, because a broker is a one-way pipe: there is nothing to read back. That changes how it behaves in several ways.

The consequences to plan for:

• Consumers must be idempotent. Because the provisioner sends events and does not confirm that they were applied, a consumer should handle a repeated message safely, using the object id and sequenceNumber.

• A new consumer that needs existing data needs a bootstrap. Messages are sent once. A consumer that starts listening later, or a queue that was purged, does not receive the earlier events. Load the initial state some other way (for example a web service or SQL export) and then apply messages from that point. See Grouper Messaging System - full sync messages for the design discussion.

• Scope selection is the main tool. As with every provisioner, use provisioning metadata on folders and groups to decide what is sent. Only in-scope objects produce messages.

Older versions of Grouper sent messages with an ESB change log consumer (EsbConsumer with EsbMessagingPublisher). The Messaging provisioner is the recommended replacement. It uses the same JSON envelope (esbEvent), so existing consumers generally keep working, and it adds:

• Granular control over which folders and groups are eligible, using the provisioning framework metadata

• Provisioner sync tracking and diagnostics in the UI

• Message content derived from what the provisioner computed, rather than raw change log entries

Configure it through the provisioner wizard in the UI (choose the Messaging "start with"), or in grouper-loader.properties / the database configuration. Properties are provisioner.<configId>.<property>.

The queue or topic is created automatically if it does not exist when loader.messaging.settings.autocreate.objects is true (the default).

Every message is a JSON envelope with a single event in the esbEvent array.

Empty values are left out of the message, which is why the examples below do not show description or parentStemId.

Entity messages use the event types MEMBER_* because they follow the ESB change log naming, where a Grouper "member" is what the provisioning framework calls an entity. A *_DELETE message has the same body as the corresponding *_ADD with a different eventType.

Sent when a group comes into scope, changes, or leaves scope.

Sent when a user comes into scope, changes, or leaves scope.

subjectSourceId and subjectIdentifier0 are taken from the entity attributes of the provisioner, so they appear when the entity data is available to the provisioner (from the subject source or the entity resolver).

Sent when a user is added to or removed from a group in scope. The membership is always reported as flattened, so an indirect member (through a subgroup) produces the same message as a direct member.

The three example messages fit together: they add group test:testGroup, then user banderson, then the membership between them, with sequenceNumber 2, 3 and 4. A consumer can join the messages using groupId and memberId.

Both run the same message-building code, so the messages are identical. What differs is how the provisioner decides what to send. Because the broker cannot be read, both compare against the provisioning sync tables, where Grouper records for each group, entity and membership whether it has been sent (in target).

A few consequences:

• A first full sync is a bootstrap. On a new provisioner nothing is recorded as sent, so a full sync sends an add message for every in-scope group, entity and membership. A large scope produces a large burst of messages, so size the queue and consumer accordingly.

• Full sync will not repair a consumer. If a message reached the broker and the consumer lost it, Grouper's record says it was sent, and a later full sync sends nothing. Recovering a consumer means replaying the data (a bootstrap) or clearing the provisioner's sync state so that everything is sent again.

• Incremental can escalate to a full sync. When a run has a large number of events (past the scoreConvertToFullSyncThreshold for the provisioner) the incremental job hands over to a full sync in the same run rather than process each event.

• Full sync updates are rare. Nothing is read back from the broker, so there are no attribute comparisons against the target to produce updates. Expect adds and deletes.

The provisioner takes the next sequenceNumber from its sync job (one counter per provisioner config id), and increments it for every object it sends. Messages are published in the order the provisioner processes them, and sequenceNumber reflects that order. Because the counter belongs to one provisioner, do not compare sequence numbers between different Messaging provisioners.

A gap in the sequence numbers can indicate a message that was lost or that has not been delivered yet. The provisioner does not resend a message that the broker accepted and later lost.

• Messages contain user identifiers (subjectId, subjectIdentifier0). Restrict access to the queue or topic accordingly.

• A change can be sent again if a run fails partway and is retried, so consumers should tolerate duplicates.

• The Grouper messaging system page lists the receive-side options, including the messaging listener daemon and the built-in grouper_message table cleanup.