Notifications (change log)

Wiki Home	Grouper Release Announcements	Grouper Guides	Grouper Deployment Guide	Community Contributions	Internal Developer Resources

Notifications

Grouper can integrate with or provision data to external systems. Approaches include:

Use the Provisioing Service Provider (PSP) to implement provisioning to a target LDAP directory based on the Change log
Use the Grouper ESB Connector or XMPP to implement notifications from the Change log
Implement a java interface to real-time notify an external system to do an incremental or full refresh based on notifications from the Change log

Change Log Events

As of Grouper 2.0, the following change log events are supported. Note that Grouper 2.1 no longer has notifications on flattened permissions due to performance concerns. Instead, whenever anything related to a permission changes (including memberships and all the hierarchies that could be involved in forming a permission), change log events are added for all the roles involved. The action name for the change log entry is permissionChangeOnRole.

This diagram shows the Change Log and Notifications in the Grouper Architecture.

Change Log Category	Action Name
attributeAssign	addAttributeAssign
attributeAssign	deleteAttributeAssign
attributeAssignAction	addAttributeAssignAction
attributeAssignAction	deleteAttributeAssignAction
attributeAssignAction	updateAttributeAssignAction
attributeAssignActionSet	addAttributeAssignActionSet
attributeAssignActionSet	deleteAttributeAssignActionSet
attributeAssignValue	addAttributeAssignValue
attributeAssignValue	deleteAttributeAssignValue
attributeDef	addAttributeDef
attributeDef	deleteAttributeDef
attributeDef	updateAttributeDef
attributeDefName	addAttributeDefName
attributeDefName	deleteAttributeDefName
attributeDefName	updateAttributeDefName
attributeDefNameSet	addAttributeDefNameSet
attributeDefNameSet	deleteAttributeDefNameSet
group	addGroup
group	deleteGroup
group	updateGroup
groupField	addGroupField
groupField	deleteGroupField
groupField	updateGroupField
groupType	addGroupType
groupType	deleteGroupType
groupType	updateGroupType
groupTypeAssignment	assignGroupType
groupTypeAssignment	unassignGroupType
member	addMember
member	changeSubject
member	deleteMember
member	updateMember
membership	addMembership
membership	deleteMembership
membership	updateMembership
permission	permissionChangeOnRole (Grouper 2.1+)
privilege	addPrivilege
privilege	deletePrivilege
privilege	updatePrivilege
roleSet	addRoleSet
roleSet	deleteRoleSet
stem	addStem
stem	deleteStem
stem	updateStem

Implementing a consumer

To implement a consumer:

1. Make sure the change log is enabled (it is by default) in the grouper.properties

# if we should insert records into grouper_change_log_temp when events happen
# defaults to true
changeLog.enabled = true

2. Enable (it is by default) the grouperLoader daemon process in the grouper-loader.properties which copies change log data from the temp table to the change log table (this is what ensures data integrity, strict ordering, unique id's, etc from disparate and concurrent grouper API sources). Note, by default this runs every minute at 10 seconds to the minute, the cron can generally be left blank unless this schedule needs to be adjusted

# should the change log temp to change log daemon run?
changeLog.changeLogTempToChangeLog.enable = true

#quartz cron-like schedule for change log temp to change log daemon, the default is 50 seconds after every minute: 50 * * * * ?
#leave blank to disable this
changeLog.changeLogTempToChangeLog.quartz.cron =

3. Extend the base class for a change log consumer. Code the notification part which sends appropriate data to external systems. In this case, it just sends the data to stdout. Note the error handling here, the keeping track of which change log sequence has been successful, the processing in a batch (currently 100 records will be sent at a time), and the coding to the change log events with constants (DONT USE string01, string02, etc)

package edu.internet2.middleware.grouper.changeLog.consumer;

import java.util.List;
import edu.internet2.middleware.grouper.changeLog.ChangeLogConsumerBase;
import edu.internet2.middleware.grouper.changeLog.ChangeLogEntry;
import edu.internet2.middleware.grouper.changeLog.ChangeLogLabels;
import edu.internet2.middleware.grouper.changeLog.ChangeLogProcessorMetadata;
import edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin;


/**
 * just print out some of the events
 */
public class PrintTest extends ChangeLogConsumerBase {

  /**
   * @see edu.internet2.middleware.grouper.changeLog.ChangeLogConsumerBase#processChangeLogEntries(java.util.List, edu.internet2.middleware.grouper.changeLog.ChangeLogProcessorMetadata)
   */
  @Override
  public long processChangeLogEntries(List<ChangeLogEntry> changeLogEntryList,
      ChangeLogProcessorMetadata changeLogProcessorMetadata) {

    long currentId = -1;

    //try catch so we can track that we made some progress
    try {
      for (ChangeLogEntry changeLogEntry : changeLogEntryList) {
        currentId = changeLogEntry.getSequenceNumber();

        //if this is a group type add action and category
        if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_TYPE_ADD)) {

          //print the name from the entry
          System.out.println("Group type add, name: "
              + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_TYPE_ADD.name));
        }

        if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_TYPE_DELETE)) {

          //print the name from the entry
          System.out.println("Group type delete, name: "
              + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_TYPE_DELETE.name));
        }

        //if this is a group add action and category
        if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_ADD)) {

          //print the name from the entry
          System.out.println("Group add, name: "
              + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_ADD.name));
        }

        if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_DELETE)) {

          //print the name from the entry
          System.out.println("Group delete, name: "
              + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_DELETE.name));
        }

        if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_UPDATE)) {

          //print the name from the entry
          System.out.println("Group update, name: "
              + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.name)
              + ", property: " + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.propertyChanged)
              + ", from: '" + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.propertyOldValue)
              + "', to: '" + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.propertyNewValue) + "'");
        }

        //we successfully processed this record
      }
    } catch (Exception e) {
      changeLogProcessorMetadata.registerProblem(e, "Error processing record", currentId);
      //we made it to this -1
      return currentId-1;
    }
    if (currentId == -1) {
      throw new RuntimeException("Couldnt process any records");
    }

    return currentId;
  }

}

4. Register this consumer in the grouper-loader.properties (note, the cron string might not be needed. If blank, it will default to on the minute, every minute [right after the temp daemon]. If there are multiple consumers, they will be staggered by 2 seconds)

#specify the consumers here.  specify the consumer name after the changeLog.consumer. part.  This example is "ldappc"
#but it could be changeLog.consumer.myConsumerName.class
#the class must extend edu.internet2.middleware.grouper.changeLog.ChangeLogConsumerBase
#changeLog.consumer.ldappc.class =

#the quartz cron is a cron-like string.  it defaults to every minute on the minute (since the temp to change log job runs
#at 10 seconds to each minute).  it defaults to this: 0 * * * * ?
#though it will stagger each one by 2 seconds
#changeLog.consumer.ldappc.quartzCron =

changeLog.consumer.printTest.class = edu.internet2.middleware.grouper.changeLog.consumer.PrintTest
changeLog.consumer.printTest.quartzCron =

5. To test this, run the loader, and a GSH shell in separate windows. Enter commands which add/delete types, and add/update/delete groups in the GSH shell, see print statements to STDOUT in the loader window. Start the loader:

C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper>bin\gsh -loader
Using GROUPER_HOME:           C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\..
Using GROUPER_CONF:           C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\../conf
Using JAVA:                   "c:\dev_inst\java/bin/java"
using MEMORY:                 64m-512m
Grouper starting up: version: 1.5.0-rc1, build date: 2009/06/10 01:09:59, env: DEV
...

6. Start up GSH, and enter some commands

C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper>bin\gsh
Using GROUPER_HOME:           C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\..
Using GROUPER_CONF:           C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\../conf
Using JAVA:                   "c:\dev_inst\java/bin/java"
using MEMORY:                 64m-512m
...
gsh 0% typeAdd("testA");
type: 'testA'
gsh 1% addRootStem("stem1", "stem1");
stem: name='stem1' displayName='stem1' uuid='ece24338-afde-4d5f-8593-d11b8b4341aa'
gsh 2% group = addGroup("stem1", "group1", "group1");
group: name='stem1:group1' displayName='stem1:group1' uuid='6ada2741-20b0-4cb6-9780-b07cfaf31db3'
gsh 3% typeDel("testA");
true
gsh 4% group.setDisplayExtension("group2");
gsh 5% group.setDescription("my description");
gsh 6% group.store();
gsh 7% typeAdd("testB");
type: 'testB'
gsh 8% typeDel("testB");
true
gsh 9% group.delete();
gsh 10%

7. Switch back to the loader window, and see the output (especially after waiting until the top of the minute, so that the daemon has run again). Note there are multiple entries for one group update since multiple properties have changed.

C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper>bin\gsh -loader
Using GROUPER_HOME:           C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\..
Using GROUPER_CONF:           C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\../conf
Using JAVA:                   "c:\dev_inst\java/bin/java"
using MEMORY:                 64m-512m
Group type add, name: testA
Group add, name: stem1:group1
Group type delete, name: testA
Group update, name: stem1:group1, property: description, from: 'null', to: 'my description'
Group update, name: stem1:group1, property: displayExtension, from: 'group1', to: 'group2'
Group type add, name: testB
Group type delete, name: testB
Group delete, name: stem1:group1

8. Note that the state of this consumer is stored in the database. So if grouper is shutdown, or the loader is shutdown, and brought back up, it will start up where it left off. The name is the name configured in the grouper-loader.properties above

select * from grouper_change_log_consumer

name       last_sequence_processed  last_updated   created_on     id                                    hibernate_version_number
--------------------------------------------------------------------------------------------------------------------------------
printTest  14                       1244611320119  1244610780084  57bd90bc-64b9-43c7-8023-dbb2b5fcbd03  4

To do

Add retry timeouts and max retries (currently it will just keep retrying every cycle of loader (every minute?) if there is an exception in target system. It should sleep for progressively longer and longer until hitting a max
Add a friendly view so it is easy to look at records
Add in other notification types: attributes, group rename, privileges, etc
Make sure effective memberships with the new membership db layout
Add a cleanup daemon which deletes old or too many records (configurable)
Add filters so that all systems are not notified on all change log entries
Add a web service to read notifications
Add hook on change log entry (pre/post insert/update/delete
Add import/export to xml

Design / FAQ

The change log is transactional. If a rollback or failure occurs in Grouper, the change log will be in sync.
If the destination is unreachable, it will retry next time with the same record
You can query the change log directly with the API through the DAO
Change log entries are ordered so that they notify in an order that will work (i.e. create a group before adding members)
There is a unique id (sequential sequence) for each row in change log
The timestamp is in micros, and will never be the same on the same jvm as another record
No database triggers are user, it's an all Java solution
There are 12 columns to stash data in the change log entry
Grouper keeps track of progress of consumers run in the loader

Page tree