Include Page | ||||
---|---|---|---|---|
|
...
The Grouper change log consists of three tables. the change log temp table is where every grouper process writes events. The change log temp to change log processes these events, gives them a sequential numeric id, calculates point in time calculations, and moves the data to the change log table. Change log consumers read from the change log table and keep a pointer to their progress in the change log consumer table.
Troubleshooting
Change log
Panel | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
This topic is discussed in the Advanced Topics training video. |
...
- Use the Provisioning Service Provider (PSP). The PSP is able to incrementally provision one or more target LDAP directories (including Active Directory) based on the Change log. It can also provision SPML targets.
- Use the Grouper ESB Connector or XMPP to implement notifications from the Change log.
- Implement your own custom change log consumer in Java. This provides additional flexibility since you can get direct access to all the information for each change, but does require custom code. This is most useful if you are provisioning a target that cannot be provisioned by the other methods.
Change Log Events
As of Grouper 2.0, the following change log events are supported. Note that Grouper 2.1 no longer has notifications on flattened permissions due to performance concerns. Instead, whenever anything related to a permission changes (including memberships and all the hierarchies that could be involved in forming a permission), change log events are added for all the roles involved. The action name for the change log entry is permissionChangeOnRole.
...
Here are some change log events, you should look in source for your version of Grouper to get the full list. See ChangeLogTypeBuiltin.java or the GROUPER_CHANGE_LOG_TYPE table.
Change Log Category | Action Name |
---|---|
attributeAssign | addAttributeAssign |
attributeAssign | deleteAttributeAssign |
attributeAssignAction | addAttributeAssignAction |
attributeAssignAction | deleteAttributeAssignAction |
attributeAssignAction | updateAttributeAssignAction |
attributeAssignActionSet | addAttributeAssignActionSet |
attributeAssignActionSet | deleteAttributeAssignActionSet |
attributeAssignValue | addAttributeAssignValue |
attributeAssignValue | deleteAttributeAssignValue |
attributeDef | addAttributeDef |
attributeDef | deleteAttributeDef |
attributeDef | updateAttributeDef |
attributeDefName | addAttributeDefName |
attributeDefName | deleteAttributeDefName |
attributeDefName | updateAttributeDefName |
attributeDefNameSet | addAttributeDefNameSet |
attributeDefNameSet | deleteAttributeDefNameSet |
group | addGroup |
group | deleteGroup |
group | updateGroup |
groupField | addGroupField |
groupField | deleteGroupField |
groupField | updateGroupField |
groupType | addGroupType |
groupType | deleteGroupType |
groupType | updateGroupType |
groupTypeAssignment | assignGroupType |
groupTypeAssignment | unassignGroupType |
member | addMember |
member | changeSubject |
member | deleteMember |
member | updateMember |
membership | addMembership |
membership | deleteMembership |
membership | updateMembership |
permission | permissionChangeOnRole (Grouper 2.1+) |
privilege | addPrivilege |
privilege | deletePrivilege |
privilege | updatePrivilege |
roleSet | addRoleSet |
roleSet | deleteRoleSet |
stem | addStem |
stem | deleteStem |
stem | updateStem |
Anchor | ||||
---|---|---|---|---|
|
Implementing
...
a consumer
...
1. Make sure the change log is enabled (it is by default) in the grouper.properties
Code Block |
---|
# if we should insert records into grouper_change_log_temp when events happen
# defaults to true
changeLog.enabled = true
|
2. Enable (it is by default) the grouperLoader daemon process in the grouper-loader.properties which copies change log data from the temp table to the change log table (this is what ensures data integrity, strict ordering, unique id's, etc from disparate and concurrent grouper API sources). Note, by default this runs every minute at 10 seconds to the minute, the cron can generally be left blank unless this schedule needs to be adjusted
...
:
...
3. Extend the base class for a change log consumer. Note, there are a lot of options here discussed below
Extend ChangeLogConsumerImpl
This is a new class in Grouper 2.3+ that allows you to create an attribute, assign it to folder(s) or group(s), and assigned groups/memberships will be provisioned.
Create and assign an attribute in GSH (note you can do this with the UI as well).
Code Block |
---|
GrouperSession grouperSession = GrouperSession.startRootSession();
AttributeDef provisioningMarkerAttributeDef = new AttributeDefSave(grouperSession).assignCreateParentStemsIfNotExist(true).assignName("attr:someAttrDef").assignToStem(true).assignToGroup(true).save();
AttributeDefName provisioningMarkerAttributeName = new AttributeDefNameSave(grouperSession, provisioningMarkerAttributeDef).assignName("attr:provisioningMarker").save()
Stem parentFolder = StemFinder.findByName(grouperSession, "some:folder", true);
parentFolder.getAttributeDelegate().assignAttribute(provisioningMarkerAttributeName);
Group someGroup = GroupFinder.findByName(grouperSession, "another:folder:someGroup", true);
someGroup .getAttributeDelegate().assignAttribute(provisioningMarkerAttributeName);
|
Configure the class in grouper-loader.properties
Code Block |
---|
changeLog.consumer.abc.class = edu.internet2.middleware.grouper.changeLog.consumer.PrintChangeLogConsumer
# note: this name matches the attribute name created in the example above
changeLog.consumer.abc.syncAttributeName = attr:provisioningMarker
changeLog.consumer.abc.quartzCron =
# defaults to true if not configured
changeLog.consumer.abc.retryOnError = true |
There are certain methods to override to sync groups and memberships
Code Block |
---|
public class SomeChangeLogConsumer extends ChangeLogConsumerBaseImpl {
protected void renameGroup(String oldGroupName, String newGroupName, ChangeLogEntry changeLogEntry) {
}
protected void removeMovedGroup(String oldGroupName, ChangeLogEntry changeLogEntry) {
}
protected void addGroup(Group group, ChangeLogEntry changeLogEntry) {
}
protected void addGroupAndMemberships(Group group, ChangeLogEntry changeLogEntry) {
}
protected void updateGroup(Group group, ChangeLogEntry changeLogEntry) {
}
protected void removeGroup(Group group, ChangeLogEntry changeLogEntry) {
}
protected void removeDeletedGroup(PITGroup pitGroup, ChangeLogEntry changeLogEntry) {
}
protected void addMembership(Subject subject, Group group, ChangeLogEntry changeLogEntry) {
}
protected void removeMembership(Subject subject, Group group, ChangeLogEntry changeLogEntry) {
}
protected boolean isFullSyncRunning(String consumerName) {
}
}
|
Extend ChangeLogConsumerBase
This is the low level change log consumer that gives you full control
Code the notification part which sends appropriate data to external systems. In this case, it just sends the data to stdout. Note the error handling here, the keeping track of which change log sequence has been successful, the processing in a batch (currently 100 records will be sent at a time), and the coding to the change log events with constants (DONT USE string01, string02, etc)
Code Block |
---|
package edu.internet2.middleware.grouper.changeLog.consumer;
import java.util.List;
import edu.internet2.middleware.grouper.changeLog.ChangeLogConsumerBase;
import edu.internet2.middleware.grouper.changeLog.ChangeLogEntry;
import edu.internet2.middleware.grouper.changeLog.ChangeLogLabels;
import edu.internet2.middleware.grouper.changeLog.ChangeLogProcessorMetadata;
import edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin;
/**
* just print out some of the events
*/
public class PrintTest extends ChangeLogConsumerBase {
/**
* @see edu.internet2.middleware.grouper.changeLog.ChangeLogConsumerBase#processChangeLogEntries(java.util.List, edu.internet2.middleware.grouper.changeLog.ChangeLogProcessorMetadata)
*/
@Override
public long processChangeLogEntries(List<ChangeLogEntry> changeLogEntryList,
ChangeLogProcessorMetadata changeLogProcessorMetadata) {
long currentId = -1;
//try catch so we can track that we made some progress
try {
for (ChangeLogEntry changeLogEntry : changeLogEntryList) {
currentId = changeLogEntry.getSequenceNumber();
//if this is a group type add action and category
if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_TYPE_ADD)) {
//print the name from the entry
System.out.println("Group type add, name: "
+ changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_TYPE_ADD.name));
}
if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_TYPE_DELETE)) {
//print the name from the entry
System.out.println("Group type delete, name: "
+ changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_TYPE_DELETE.name));
}
//if this is a group add action and category
if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_ADD)) {
//print the name from the entry
System.out.println("Group add, name: "
+ changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_ADD.name));
}
if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_DELETE)) {
//print the name from the entry
System.out.println("Group delete, name: "
+ changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_DELETE.name));
}
if (changeLogEntry.equalsCategoryAndAction(ChangeLogTypeBuiltin.GROUP_UPDATE)) {
//print the name from the entry
System.out.println("Group update, name: "
+ changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.name)
+ ", property: " + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.propertyChanged)
+ ", from: '" + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.propertyOldValue)
+ "', to: '" + changeLogEntry.retrieveValueForLabel(ChangeLogLabels.GROUP_UPDATE.propertyNewValue) + "'");
}
//we successfully processed this record
}
} catch (Exception e) {
changeLogProcessorMetadata.registerProblem(e, "Error processing record", currentId);
//we made it to this -1
return currentId-1;
}
if (currentId == -1) {
throw new RuntimeException("Couldnt process any records");
}
return currentId;
}
}
|
Compile and jar this up
Code Block |
---|
You need the grouper API jars, you can get those from a WS or UI deployment, or from the API itself...
In this case I used my WS deployment:
[appadmin@fasttest-small-b-01 temp]$ pwd
/opt/appserv/tomcat/apps/grouperWs/webapps/grouperWs/WEB-INF/temp
[appadmin@fasttest-small-b-01 temp]$ mkdir -p edu/internet2/middleware/grouper/changeLog/consumer
... put the source in the file with your favorite editor ...
[appadmin@fasttest-small-b-01 temp]$ cat edu/internet2/middleware/grouper/changeLog/consumer/PrintTest.java
package edu.internet2.middleware.grouper.changeLog.consumer;
import java.util.List;
import edu.internet2.middleware.grouper.changeLog.ChangeLogConsumerBase;
import edu.internet2.middleware.grouper.changeLog.ChangeLogEntry;
import edu.internet2.middleware.grouper.changeLog.ChangeLogLabels;
import edu.internet2.middleware.grouper.changeLog.ChangeLogProcessorMetadata;
import edu.internet2.middleware.grouper.changeLog.ChangeLogTypeBuiltin;
/**
* just print out some of the events
*/
public class PrintTest extends ChangeLogConsumerBase {
...
[appadmin@fastprod-medium-a-01 temp]$ javac -classpath "../lib/*" -sourcepath . edu/internet2/middleware/grouper/changeLog/consumer/PrintTest.java
Then you can put that in a jar, put the jar in the grouper lib dir, should be good to go:
[appadmin@fastprod-medium-a-01 temp]$ jar cvf printTest.jar .
added manifest
adding: edu/(in = 0) (out= 0)(stored 0%)
adding: edu/internet2/(in = 0) (out= 0)(stored 0%)
adding: edu/internet2/middleware/(in = 0) (out= 0)(stored 0%)
adding: edu/internet2/middleware/grouper/(in = 0) (out= 0)(stored 0%)
adding: edu/internet2/middleware/grouper/changeLog/(in = 0) (out= 0)(stored 0%)
adding: edu/internet2/middleware/grouper/changeLog/consumer/(in = 0) (out= 0)(stored 0%)
adding: edu/internet2/middleware/grouper/changeLog/consumer/PrintTest.java(in = 3289) (out= 793)(deflated 75%)
adding: edu/internet2/middleware/grouper/changeLog/consumer/PrintTest.class(in = 3724) (out= 1485)(deflated 60%)
[appadmin@fastprod-medium-a-01 temp]$ ls -altr
total 16
drwxr-xr-x. 9 appadmin users 4096 Sep 24 14:40 ..
drwxr-xr-x. 3 appadmin users 4096 Sep 24 14:41 edu
drwxr-xr-x. 3 appadmin users 4096 Sep 24 14:43 .
-rw-r--r--. 1 appadmin users 3871 Sep 24 14:43 printTest.jar
[appadmin@fastprod-medium-a-01 temp]$ jar tf printTest.jar
META-INF/
META-INF/MANIFEST.MF
edu/
edu/internet2/
edu/internet2/middleware/
edu/internet2/middleware/grouper/
edu/internet2/middleware/grouper/changeLog/
edu/internet2/middleware/grouper/changeLog/consumer/
edu/internet2/middleware/grouper/changeLog/consumer/PrintTest.java
edu/internet2/middleware/grouper/changeLog/consumer/PrintTest.class
[appadmin@fastprod-medium-a-01 temp]$
|
4. Register this consumer in the grouper-loader.properties (note, the cron string might not be needed. If blank, it will default to on the minute, every minute [right after the temp daemon]. If there are multiple consumers, they will be staggered by 2 seconds)
Code Block |
---|
#specify the consumers here. specify the consumer name after the changeLog.consumer. part. This example is "ldappc"
#but it could be changeLog.consumer.myConsumerName.class
#the class must extend edu.internet2.middleware.grouper.changeLog.ChangeLogConsumerBase
#changeLog.consumer.ldappc.class =
#the quartz cron is a cron-like string. it defaults to every minute on the minute (since the temp to change log job runs
#at 10 seconds to each minute). it defaults to this: 0 * * * * ?
#though it will stagger each one by 2 seconds
#changeLog.consumer.ldappc.quartzCron =
changeLog.consumer.printTest.class = edu.internet2.middleware.grouper.changeLog.consumer.PrintTest
changeLog.consumer.printTest.quartzCron =
|
5. To test this, run the loader, and a GSH shell in separate windows. Enter commands which add/delete types, and add/update/delete groups in the GSH shell, see print statements to STDOUT in the loader window. Start the loader:
Code Block |
---|
C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper>bin\gsh -loader
Using GROUPER_HOME: C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\..
Using GROUPER_CONF: C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\../conf
Using JAVA: "c:\dev_inst\java/bin/java"
using MEMORY: 64m-512m
Grouper starting up: version: 1.5.0-rc1, build date: 2009/06/10 01:09:59, env: DEV
...
|
6. Start up GSH, and enter some commands
Code Block |
---|
C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper>bin\gsh
Using GROUPER_HOME: C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\..
Using GROUPER_CONF: C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\../conf
Using JAVA: "c:\dev_inst\java/bin/java"
using MEMORY: 64m-512m
...
gsh 0% typeAdd("testA");
type: 'testA'
gsh 1% addRootStem("stem1", "stem1");
stem: name='stem1' displayName='stem1' uuid='ece24338-afde-4d5f-8593-d11b8b4341aa'
gsh 2% group = addGroup("stem1", "group1", "group1");
group: name='stem1:group1' displayName='stem1:group1' uuid='6ada2741-20b0-4cb6-9780-b07cfaf31db3'
gsh 3% typeDel("testA");
true
gsh 4% group.setDisplayExtension("group2");
gsh 5% group.setDescription("my description");
gsh 6% group.store();
gsh 7% typeAdd("testB");
type: 'testB'
gsh 8% typeDel("testB");
true
gsh 9% group.delete();
gsh 10%
|
7. Switch back to the loader window, and see the output (especially after waiting until the top of the minute, so that the daemon has run again). Note there are multiple entries for one group update since multiple properties have changed.
Code Block |
---|
C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper>bin\gsh -loader
Using GROUPER_HOME: C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\..
Using GROUPER_CONF: C:\mchyzer\isc\dev\grouper-qs-1.2.0\grouper\bin\../conf
Using JAVA: "c:\dev_inst\java/bin/java"
using MEMORY: 64m-512m
Group type add, name: testA
Group add, name: stem1:group1
Group type delete, name: testA
Group update, name: stem1:group1, property: description, from: 'null', to: 'my description'
Group update, name: stem1:group1, property: displayExtension, from: 'group1', to: 'group2'
Group type add, name: testB
Group type delete, name: testB
Group delete, name: stem1:group1
|
8. Note that the state of this consumer is stored in the database. So if grouper is shutdown, or the loader is shutdown, and brought back up, it will start up where it left off. The name is the name configured in the grouper-loader.properties above
...
you should impement an EsbEventListener (layer on top), not a change log consumer
SQL View
There is a friendly SQL view: grouper_change_log_entry_v which will be more friendly to query if you are debugging something than the grouper_change_log_entry table
To do
- Add retry timeouts and max retries (currently it will just keep retrying every cycle of loader (every minute?) if there is an exception in target system. It should sleep for progressively longer and longer until hitting a max
- Add filters so that all systems are not notified on all change log entries
- Add hook on change log entry (pre/post insert/update/delete
- Add import/export to xml
Design / FAQ
- The change log is transactional. If a rollback or failure occurs in Grouper, the change log will be in sync.
- If the destination is unreachable, it will retry next time with the same record
- You can query the change log directly with the API through the DAO
- Change log entries are ordered so that they notify in an order that will work (i.e. create a group before adding members)
- There is a unique id (sequential sequence) for each row in change log
- The timestamp is in micros, and will never be the same on the same jvm as another record
- No database triggers are user, it's an all Java solution
- There are 12 columns to stash data in the change log entry
- Grouper keeps track of progress of consumers run in the loader