Configuration

Threads

loader.incrementalThreads=true
loader.incrementalThreadPoolSize=10

Jobs

otherJob.incrementalLoader1.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob
otherJob.incrementalLoader1.quartzCron = 0 * * * * ?
otherJob.incrementalLoader1.databaseName=warehouse
otherJob.incrementalLoader1.tableName=myincrementaltable
 
otherJob.incrementalLoader2.class = edu.internet2.middleware.grouper.app.loader.GrouperLoaderIncrementalJob
otherJob.incrementalLoader2.quartzCron = 0 * * * * ?
otherJob.incrementalLoader2.databaseName=warehouse2
otherJob.incrementalLoader2.tableName=myincrementaltable

Database setup

Add triggers on your existing tables that are used by the loader jobs such that new rows would be added to another table (e.g. myincrementaltable) when a user's membership changes (or potentially changes) for a loader job.  The table has the following columns:

• subject_id
• subject_identifier - mutually exclusive with subject_id
• source_id - optional
• loaderGroupName - Grouper group name of a group defined as SQL_SIMPLE or SQL_GROUP_LIST.
• timestamp - milliseconds

Daemon

• The daemon would retrieve the 1000 oldest rows, removing duplicates.
• Spawn off threads based on the thread configuration.
• If SQL_SIMPLE, query grouperLoaderQuery to see if user should be a member.  Query and change Grouper if needed.
• If SQL_GROUP_LIST, query grouperLoaderQuery to see which groups the user should be a member of.  Need to figure out which Grouper groups are managed by this loader job to see what changes to make.  Get the distinct group names from grouperLoaderGroupQuery if available, otherwise grouperLoaderQuery.  And see if the user is a member of each.  This is clearly not efficient.
• Need to take into account grouperLoaderAndGroups.  If configured, user must be in all all of the "and" groups to be in the loader group.
• Any special logic around include/excludes?
• If group doesn't exist in Grouper, ignore.
• Fail safe?
• Delete rows from table when finished.
• Log to grouper_loader_log.