Child pages
  • Grouper Call 21-July-2010
Skip to end of metadata
Go to start of metadata

Grouper Call 21-July-2010

*Attending*

Chris Hyzer, U. Penn  (stand-in chair)
Shilen Patel, Duke 
Gary Bristol, Bristol
Jim Fox, University of Washington
Tom Zeller, U. Memphis  
Steve Olshansky, Internet2  
Emily Eisbruch, Internet2 (scribe) 

*Action Items*

*New Action Items*

[GrouperWG:AI] (Shilen) will develop a test case for groups with thousands of members and add it to the test suite.
[GrouperWG:AI] (Shilen) will update the JIRA issue for GRP-458 (Manage Groups in UI should perform well when user is admin of many groups)https://bugs.internet2.edu/jira/browse/GRP-458

*Carry Over Action Items*

[GrouperWG:AI] (Jim) will do performance testing to help evaluate potential issues with the flattened memberships table.
[GrouperWG:AI] (Rob) will look at issues relating to testing the ESB Connector.

*Discussion*

*Error Deleting a Group*

Jim had reported a problem on the email list with deleting a group, and this problem is continuing.

https://lists.internet2.edu/sympa/arc/grouper-dev/2010-07/msg00019.html

Shilen stated that when a large group with  is deleted, there will be a lot of immediate membership deletions involved (including a change log entry). This is all part of one transaction, and it could possibly cause problems if the database gets so big it can't hold all that data.

Jim noted the problem occurs in deleting groups with a few thousand members.  If the cause is in fact that the database gets too big, then what will happen with deleting groups with 100,000 members?

It was agreed that in terms of stress on the database the operation really should not cause trouble.

To understand the issue further, Jim will try deleting each member individually (using a batch method) and then deleting the group, to see if the problem still occurs.

Other things Jim may try:
- Increase the shared buffer space
- Check to see if there are background processing causing trouble.

It was suggested that the Grouper test suite should include a test for groups with thousands of members. It could take a while to run, so it should be an optional test.

[GrouperWG:AI] (Shilen) will develop a test case for groups with thousands of members and add it to the test suite.

*Early Experience with Grouper 1.6*

Jim reported that Grouper 1.6 is working well at U. Washington aside from the delete group issue.

*Schedule and Scope of 1.6.1*

Targets for Grouper 1.6.1

- Code freeze at end of Sunday, Aug 1.
- Do light testing Aug. 2 and Aug. 3
- Release on Wed. Aug. 4

Brief review of some of a few of the Jira issues to be included:

GRP 469: This Jira will provide another check on the registry, which will be helpful at Penn:https://bugs.internet2.edu/jira/browse/GRP-469

GRP 466: "Put the username using UI/WS in log message from threadlocal"
Gary will work on this.https://bugs.internet2.edu/jira/browse/GRP-466

GRP 459: "postgres might not drop all views, we should check the order here or something... (cascade will drop dependent views)"  It was suggested to eliminate the dependence for now if we can't otherwise fix the problem.https://bugs.internet2.edu/jira/browse/GRP-469

GRP 443: "If there is an exception when copying / moving stems/groups the user does not see an appropriate error message"  Gary  is working on this. It might be irreproducible.

GRP 458: "Manage group in UI not performing well in user admin of many groups."
This JIRA issue is more substantial than the others. Problem is that if a person is an admin of 100,000 groups, the query will return 100,000 results. Also, for every membership, it runs Get-Groups and that causes an additional performance issue. Gary noted that we tried an alternative approach several years ago. But it didn't work very well.
Suggestion for a temporary solution : Do a count quickly to see how many groups an admin has. Then have a different behavior if an admin has many groups.

The approach was discussed of just displaying folders or just stems for a person who is an admin of many groups. But there are still issues if there is a hierarchy with much nesting.

[GrouperWG:AI] (Shilen) will update the JIRA issue for GRP-458 (Manage Groups in UI should perform well when user is admin of many groups )

*Performance and Flattened Tables*

Shilen's performance testing found that if a group with 66,000 members is added to another group, it takes about 6 minutes for the change log daemon to process.

It was agreed that this performance could be problematic in certain environments, including with the usage scenarios at U. Washington.
Jim stated that one supposed advantage of using the flattened table is to quickly get all groups someone is part of (to get a person's effective memberships). But Jim has found that people typically don't want to know just a list of what groups they are a member of. They want to know the path by which they got to be a member, and the flattened table will not help with that.

Another reason the flattened table was created was for point in time audit. There has not yet been a big need for that at U. Washington, though that could change in the future.

A batch approach for flat membership updates (GRP 477) will be in Grouper 1.6.1 and may help performance somewhat.

https://bugs.internet2.edu/jira/browse/GRP-477

Still, it probably does not make sense to expand the use of flattened tables to handling of Roles and Permissions in Grouper v 2.0. With roles and permissions the size of tables could multiply quickly.

Chris suggested approaches for handling the database for permissions:

1. Uncouple the data for roles and for permissions so we don't store each permission for each member

2. Use a dynamic role approach, where people who have a certain attribute will be assigned to a role when they log in

Next Meeting: Wed. Aug 4 at noon ET

  • No labels