USDU delete subjects after unresolvable for X days
There are several improvements in USDU for a patch in 2.4.0 API.
Attest unresolvables
After you get the patches that have better daemon and UI subject resolution, you should setup attestation for an admin to review it
New attributes
Attributes on members. These will get updated during USDU in readonly or readwrite mode.
Attribute definitions
Definition | Assigned To | Purpose | Value | Cardinality |
|---|---|---|---|---|
subjectResolutionDef | member | marker on member | marker | Single assign |
subjectResolutionValueDef | member assignment | name/value pairs | string | Single assign, single valued |
Attribute names
Name | Definition | Value |
|---|---|---|
subjectResolutionMarker | subjectResolutionDef | <none> (assigned to unresolvable subjects) |
subjectResolutionResolvable | subjectResolutionValueDef | false if this subject is currently unresolvable (as of last check). Note as of 2.5.30, this attribute has been moved to the grouper_members table as a separate column. |
subjectResolutionDateLastResolved | subjectResolutionValueDef | yyyy/mm/dd If this subject has a date and is unresolveable, leave it. if this subject doesnt have a date, and is unresolvable, then set to currentDate. |
subjectResolutionDaysUnresolved | subjectResolutionValueDef | 7 - the number of days from current date minus dateLastResolved. |
subjectResolutionLastChecked | subjectResolutionValueDef | yyyy/mm/dd the date this subject was last checked. When the USDU runs, if this subject is current unresolvable, then set to currentDate |
subjectResolutionDeleted | subjectResolutionValueDef | true when this subject is marked as deleted. All the memberships are removed at this point. Values from subjectResolutionResolvable, subjectResolutionDaysUnresolved and subjectResolutionDateLastChecked are also cleared at this point. Note as of 2.5.30, this attribute has been moved to the grouper_members table as a separate column. |
subjectResolutionDeleteDate | subjectResolutionValueDef | time when this subject is marked as deleted. |
Change USDU so that instead of deleting unresolvables now, it waits until after X days
grouper.base.properties
# global across all sources: Don't do anything if more than this number of unresolvable subjects are found
# {valueType: "integer", required: true}
usdu.failsafe.maxUnresolvableSubjects = 500
# global across all sources: if the first X subjects should be removed but stop after that limit: usdu.failsafe.maxUnresolvableSubjects
usdu.failsafe.removeUpToFailsafe = false
# global across all sources: only delete unresolvables if unresolvable for 30 days. false or 0 means remove now
# {valueType: "integer", required: false}
usdu.delete.ifAfterDays = 30
# local to one source supersedes the global settings: source ID
# {valueType: "string", required: true, regex: "^usdu\\.source\\.([^.]+)\\.sourceId$"}
# usdu.source.someLabel.sourceId = someSourceId
# local to one source supersedes the global settings: Don't do anything if more than this number of unresolvable subjects are found
# {valueType: "integer", required: true, regex: "^usdu\\.source\\.([^.]+)\\.failsafe\\.maxUnresolvableSubjects$"}
# usdu.source.someLabel.failsafe.maxUnresolvableSubjects = 500
# local to one source supersedes the global settings: if the first X subjects should be removed but stop after that limit: usdu.failsafe.maxUnresolvableSubjects
# {valueType: "integer", required: true, regex: "^usdu\\.source\\.([^.]+)\\.failsafe\\.removeUpToFailsafe$"}
# usdu.source.someLabel.failsafe.removeUpToFailsafe = false
# local to one source supersedes the global settings: only delete unresolvables if unresolvable for 30 days. false or 0 means remove now
# {valueType: "integer", required: true, regex: "^usdu\\.source\\.([^.]+)\\.delete\\.ifAfterDays$"}
# usdu.source.someLabel.delete.ifAfterDays = 30
Currently there is a readonly mode. There is a mode that deletes unresolvables. This new option will adjust the "delete unresolvables" but the unresolvables need to be unresolvable for X days.
Make sure subject deletion is audited. When a subject becomes unresolvable, add an audit. When a subject goes from unresolvable to resolvable, add an audit. When a subject is deleted, add an audit.
USDU as daemon
Currently USDU is invoked in the Grouper report (readonly), and on the command line. USDU should no longer run during the grouper report. USDU will now run an an "otherjob" configured every saturday at 1am (in grouper-loader.base.properties). The mode will be read/write or will consider the "ifAfterDays"
USDU on UI
Under Miscellaneous on UI, for grouperSysAdmins, show a "Subject resolution" link.
On that page, show stats on subject resolution. How many unresolvables per source.
Allow subjects to be searched. Show if they are resolvable or unresolvable.
Allow unresolvables to be shown in a paged list of 200. Show a table with when last resolved, when last checked, days unresolved, and when they will be deleted
Show audits for subject deletion (paged list of 200)
Have a link to the daemon log screen for this daemon
Screenshots
The idea of delaying for at least a few days means the failsafe number should be set to be large. Especially because a class of students could be many thousands to remove at once. If one sets the failsafe to be 2000 then you would only remove 2000 and then the next run would do the next 2000 and so on? If so, then this is a reasonable behavior. If hitting the failsafe means nobody gets removed - that might be problematic. Maybe “removeUpToFailsafe” is an option to be set to control which behavior is desirable?
As for the tracking attributes. You only need to track those who have come into the unresolvable state. If this is true, then this keeps the book-keeping to a minimum. I wouldn’t want to see dates and true/false being set for every subject every night. that’s a lot of unnecessary extra work.
Lastly, would it be the case the case that as a subject lookup is "not found" in normal grouper operations then this is when you would apply the attributes to the subject for unresolvable? If you did this, then there wouldn't be the need to scan the entire subject space to figure this out. The USDU processing would just become a matter of searching for attributes on subjects and operating on the candidates for USDU processing.