Attending
- Chris Hyzer, Penn, Chair
- Shilen Patel, Duke
- Chad Redman, University of North Carolina Chapel Hill
- Carey Black, the Ohio State University
- Vivek Sachdiva, independent
- Jeffrey Crawford, UCLA
- Keith Hazelton, Internet2
- Chris Hubing, Internet2
- Emily Eisbruch, Internet2
Discussion
- https://internet2.edu/community/about-us/policies/internet2-intellectual-property-policy/
- Approve minutes
- Review AIs Grouper Project Action Items (Google Doc)
- Agenda bash
Grouper Training Online Feb 9-12, 2021 - went well
Current work
Vivek
- GSH Templates
- Allows Grouper users to run arbitrary GSH scripts
- Hope to externalize with web services
- Must configure template same way
- Text will go back to web service
- Have daemons that run GSH
- Running each GSH line as its own autonomous statement
- Will change how Grouper runs a script so it works same way
- Figuring out what variables are available for the script.
- There’s an output object to print lines of output back to caller
- Can have from field output
- Script might do some validation and end early
- Can't just return from GSH so added a method
- There will a wiki explaining how to handle scripts
- How to specify output
- Discussion on Slack on support for web service types going forward
- Will support JSON REST for new operations
- This was Chris Hyzer's note on Slack:
- “Seems like we might be ready to move in the direction of REST/JSON for web services. i.e. keep supporting all the backwards compatible stuff (SOAP, REST/XML, REST/XHTML) we already have, but new services or new options to services will be REST/JSON only. and by "REST", I mean our current HTTP/POJ formats.
Also we can more fully support SCIM as a service endpoint which is more restful and generic but not grouper specific. We will also migrate grouper client to REST/JSON (instead of REST/XML) for examples. ”
- Keep other approaches in place for what already exists
- Chris does not like REST response codes
- An http code could help
- GSH has built in commands that are not very useful
- Starting point for a handful of APIs
- Don’t want things to stop compiling
- Save classes, stem save,
- Method chaining
- You won’t add a parameter
- Finder classes, Grouper Finder , Stem finder
- We will add more find classes and save classes for these scripts
- Examples: dealing w memberships, privileges, attestation
- Chris Hubing: the use case for Internet2
- Want Internet2 staff or a community members to be able to spin up working groups
- And do operations thru the COmanage user interface
- Spin up a new working group and associated services for that
- Creating a group, adding a member to the group, attestation and privileges
- Arbitrary attributes….
- Example of attribute : Group in sympa needs to be moderated
- Chris Hubing is working on a taxonomy around that based on the downstream applications
- Perhaps attestation every 365 days… for the working group
- Down the road, integrate with MidPoint, MidPoint integrates w Slack
- Next tasks for Vivek
- Work on web service wrapper
- View button may not be critical path
- Chris Hyzer and Vivek will talk about what’s needed for the Internet2 use case
Keith Hazelton:
- Interested in API approach
- Hope to tell people if you don’t have Grouper, but do have some other group system, here is what you do for the Trusted Access Platform
- Keith may create a write-up on this
More discussion
- Vivek and Chris Hyzer are hoping to finish the script work by end of weekend and create release.
- Will there be conditionals in the template?
- Chris Hubing: yes
- Chris Hubing may consult w Chris Hyzer on writing script
- Running a GSH from JAVA now runs it wrong. Will change it to how Grouper Shell runs
- Why do we need externalized labels and descriptions?
- Same person setting up template is setting up the config.
- ChrisHyzer: externalized text file has all the text for the UI,
- Consistency
- Could have trick where form shows text and puts it in externalized text file
- Internationalization issues
- Text and config files
- Fall back to externalized text in config
- Shilen: have the form elements on page and put on externalized file in database
- Hard to configure?
- In the JSON config say “This is an externalized text link”
- Chad: need to export two places when you export the config
- Must remember when you change a field name in form, also change it in config
- Decision: Think about putting externalized Text file in database
- Drop-down field type.. Is it hard coded?
- Look for groups in a certain tree for example
- Chris Hyzer : that is in the the plans
- Set a provisioner with one line, could be used in GSH templates
- Security : Only Admins can get in to write GSH script
- Create a super group of admins for GSH Scripts
Shilen:
- Duke has some Grouper admins who should not have access to this GSH template script feature
- Chris Hyzer and Vivek will work on this
- Create a group of template managers or super admins
- Could send a notification if someone gets added to the template managers group
- Is there an option to decide if this runs with a certain transaction?
- If there is a failure partway through
- Makes it easier to retry
- Do this in a transaction, will default to true
- Yes, will implement this
- Don’t want a partial template to run
Jeffrey C., UCLA
- Challenge onsite keeping computer clocks in sync
- Not Grouper's fault
- Creating groups and assigning memberships
- Change log got add a member to group before it got create group
- Change log got locked
- Instead of using the timestamp of the node? , use timestamp based on database time
- It was hard to find out what happened.
- Found a page Chris Hyzer had written that provided clues
- Had 28 groups impacted
- VMware should sync to time servers?
- Changelog goes by time
- Getting changelog events correctly ordered, using SQL clock
- Can do in changelog temp
- Or use a database specific sequence
- Worry about long running transaction
- Assigning the time to that is better than doing a sequence
- Database clock , get a function
- Issue w views and how they are concatenated
- Have grouper database function library
- So all views would not be same?
- Inserts w hibernate
- Just call a function in database
- Opportunity to add more than just functions and views
- Outside of hibernate, writing an application you want to default to current timestamp
- It fills in a SQL level
- Have timestamp column and don’t set it
- Need to capture this and see what’s possible
- Not sure the priority if Jeffrey C is all set
- For Oracle
- Put the solution into Slack
- Query goes thru and finds create group, finds some other operation earlier
- Creates an update statement,
- Concat
- Update statement
- Will give a series of commands,
- Can fix in SSL or Shib
- Harder to fix in Grouper
- If JVM detects time is off in database, just exit?
- Get current time from database and compare.
- Get hibernate to set time on a column..
Chris Hyzer
- U. Arizona wants to sync folders from one Grouper to another Grouper using SQL
- Wish the loader was more modular
- New package Sync to Grouper
- Behavior class
- Sync groups, stems, composites
- Go get from SQL, beans, Grouper
- Or from another Grouper
- Syncing composites from a database
- Simple setting “I want to sync from another Grouper”
- Carey: UUIDs, could have a membership save
- Attributes on memberships
- Move between MYSQL and other databases
- Will this approach work?
- Yes that’s what AZ is doing.
- Better to do a cutover
- This does not set the create date, Does do UUID
- Issue of downtime involved
- Stopping the database
- This seems more in flight
- This work can evolve
- It’s intended for “I set up in test, want to move to prod”
- Consider a Grouper version where we redo the database
- Databases not good at managing UUID
- Make your own UUIDs
- Here is Chris Hyzer’s Slack note about this
- We already have an integer id on several of the major tables . And we have a way to do this in a database independent way! id_index.
So what if the id_index (int?) were used for primary keys and foreign keys and we do something better for uuids (store as binary?
try to have a more deterministic prefix?) it would take a lot of effort, but it is worth considering...
We could mock up a bunch of groups and memberships both ways and see if there is a noticeable gain before we go down that path...
The attribute_assign table has 10 uuid cols out of 20.
The membership table has 10 uuid cols out of 16.
The group_set table has 18 uuid cols out of 22.
this design decision pre-dates everyone on the team including me, but I guess its never too late to revisit stuff
Chad
- Working on properties
- Config values are missing
- Tracking down
- Updated JIRA
- Script work: UNC can be testers
- Chris: flesh out APIs , method chaining pattern
- Attestations
- Chad work on this?
- For Azure, Chad can look at it
- Can Azure provision teams?
- Another API call after a unified group in Azure
- Need to be able to mark as unified or not
- Other things needing support from Azure
- Welcome message, may want to turn off in some cases
- Carey: Good use case for a GSH script template?
- Change log consumer supports Jexel for name, description, Email ,
- If we can’t support it, figure out what to do for people depending on it
- Chris and Chad will chat about the gaps around Azure
Shilen
- Membership finder changes
- Able to sort and page using membership data
- Sort based on membership time
- Make max page size configurable
- Local entities weren’t being sorted properly
- Fixed
- Provisioning work:
- If you had one config for provisioning that was invalid, the daemon would delete all provisioning configs, it was calling attribute assign finder
- It’s about marking things as configurable
- Switched provisioning to store config ID instead of true false
- Will change how propagation happens
- Main problem, the provisionable stuff , when it propagates,
- The daemon was inefficient and incorrect
- Looking at the types and other things that propagate
Issue Roundup
Jiras in past two weeks
grouper custom templates via gsh
AttributeDef trace privileges links back to subject page, not attributeDef page
sync grouper from another grouper via sql
when looking at provisioner, have edit button (other buttons too?)
entity group link error should not appear
loader queries should be able to start with "with"
allow loader queries to be longer than 4k
Local entities should sort by display extension instead of name by default
MembershipFinder paging/sorting by membership fields
subject wizard ldap search subject scope has one option
provisioning metadata should not be assigned if no metadata there
deleting an incremental daemon doesnt delete that daemon
if you edit a provisioning daemon config it doesnt show the provisioner config id
grouper instrumentation needs to clear out old server names
ability to “run loader diagnostics” processes in a CI/CD pipeline
Allow MembershpFinder max pageSize to be configurable
Better error message for users not allowed to login to Grouper UI
client should show usage if no usage file on classpath
Issue with daemon that deletes invalid provisioning assignments
folder privs should not show inherited attribute read/update if subject has create
provisioningDoProvision should store the provisionerId if provisionable instead of true/false
view provisioning on subject throws grouper session error
add ldap loader filter (with cache)
add ldap loader lookup by dn (with cache)
config refresh doesnt work, casts to int when should be long
Ability to limit provisioning to specific targets by group for WS
check certain source for registry subjects after adding (default "jdbc")
provisioning incrementals finds multiple subjects with same matching id, but they are the same
convertDnToSpecificValue does not unescape commas
daemon configuration for reports
grouperExternal subject source gets removed
grouper sqs should take in region
UI function to union/complement an id list
Grouper Emails in past two weeks
- [grouper-users] Grouper slack channel, Leo, Michael, 02/04/2021
- Re: [grouper-users] Grouper slack channel, Emily Eisbruch, 02/17/2021
- [grouper-users] Grouper Installation error, [Malathi] [D]\", 02/08/2021
- [grouper-users] Install Grouper, [Malathi] [D]\", 02/08/2021
- [grouper-users] Is there any way to search folders or groups with certain attributes (eg. provision_to) in grouper 2.4 UI., Siju Jacob, 02/11/2021
- <Possible follow-up(s)>
- Re: [grouper-users] Is there any way to search folders or groups with certain attributes (eg. provision_to) in grouper 2.4 UI., Black, Carey M., 02/12/2021
- [grouper-users] LDAP over SSL error - CertificateException, Al Lilianstrom, 02/15/2021
- Re: [grouper-users] LDAP over SSL error - CertificateException, Al Lilianstrom, 02/15/2021
- Re: [grouper-users] LDAP over SSL error - CertificateException, Hyzer, Chris, 02/15/2021
Grouper wiki updates in past two weeks
- Install the Grouper v2.5 container with maturity level 0 manually
- Grouper custom template via GSH
- Grouper LDAP provisioner in v2.5
- Grouper custom template via GSH
- Grouper reporting
- Grouper forms, workflow and approvals
- Grouper LDAP provisioner v2.5 use case Michigan
- Grouper provisioning strategy
- Grouper provisioning: identifying groups for provisioning
- Grouper Loader LDAP bulk lookup by filter (with cache)
- Grouper Loader LDAP simple lookup by filter (with cache)
- Grouper Loader LDAP simple lookup by DN (with cache)
- Grouper Loader LDAP simple lookup by DN (no cache)
- Grouper Loader LDAP simple from hasMember
- Grouper Loader LDAP dereference example convert DN to RDN value
- Grouper attribute framework queries
- Grouper subject source configuration wizard
- v2.5 Release Notes
- Grouper attributes
- Provisioning Models
Grouper Slack in past two weeks
Jeffrey C is there a per loader job failsafe limit available? I found a global setting.
Jeffrey C Most of our campus uses MSSQL servers, so we are forced to use them as a source for loader jobs. today one of the DB’s was taken offline late because a processes that feeds it data took unusually long. the loader didn’t seem to think it was an error that our user was kicked off and just happily removed a bunch of users since they were no longer in the loaded data.
I’m not sure how MSSQL reacts when a DB is taken down but I think it would have generated an error that I don’t think the grouper loader processes detected.
Paul R anyone have experience with the aws sqs messaging in grouper 2.4.
Ryan R when new attributes are added to the subject.properties file what needs to happen in Grouper for the new attribute to be populated?
d
Liam Is it possible to run a membership audit report on a group of groups (composite or otherwise), and see the changes in the member groups? Or if you want the changes in those member groups, do you have to go all the way down to the original children?
Erik topic of membership math, which Grouper database memberships views do what? Is there any documentation on how these are configured, and what are the differences (_v vs _all_v) ?
Liam Do I need to worry about this? It comes up every time I go into the Daemon jobs..
Error: can’t find daemon config for jobName grouper_report_9d681135d4084d0ab06b992e47615c4b_71c5e29a75764ddc8b4ca2f6544738ff
Erik I've got several working LDAP loader jobs, I want to create an LDAP_GROUP_SIMPLE to slurp an AD group membership into a Grouper group.
Carey GRP-1747 Group 2.3 Function (UI button) --> View Audit Log SQL timeout
Carey I have an app that has multiple folder structures ( dev, test, prod ) and they would like to “review the environments” for deltas in Grouper privileges, missing groups, extra groups, etc…
Chris H Seems like we might be ready to move in the direction of REST/JSON for web services.
Justin After the 2.5 upgrade, we’ve been trying to work through the Duo connector.
Liam anyone playing with the new provisioner yet? Is it possible to only provision the policy groups from a policy structure that’s been applied to the application structure,
Liam If the provisioner throws this error…
Why do multiple entities have the same matching id???..
Liam Is there a grouper provisioner for Dropbox?
Erik When I export a group in the UI (2.5.33), even if I say export "All Member Data", I am not able to pull out their subjectIdentifier or any other attributes that I made available via the subject.properties.
Carey I keep seeing this in my daily Grouper report… but I am not sure how to track it down. Any clues?
Liam trying to run an incremental provisioner using the new provisioning framework, and I’m getting an INAPPROPRIATE_MATCHING error…
Liam Does the SCIM 2 provisioner work yet?
jasonrap Is anyone running Grouper in Azure AKS?
Sudheer Are these properties used to grant access to non-admin users to configure loader jobs?
Sudheer I’m getting this error when I try to configure a loader job by logging in into grouper as non-grouper admin user:
Erik The way I understand the design, loader jobs are best left for the Grouper Admins, because they run as "root" and people could create loaders that impact folders they may not have access to. Basically, it's all-or-nothing with the privileges.
Mike P Was looking at the Grouper database and was checking out grouper_change_log_entry. Each entry has a change_log_type_id which points to one of grouper_change_log_type. Why is a 40 char key used for this? Wouldn’t an int suffice and save the rather massive indexes?
Jeffrey C Our test grouper environment seems to have gotten the CHANGE_LOG_changeLogTempToChangeLog stuck with an error. I’ll put the error in a thread. This may be related to an obliterateStem operation
Carey
GRP-3137 attestation email content (body) should be able to
Chris H
Some loader LDAP documentation / enhancements:
Andy M I've got some SQL for a loader job that begins with a "WITH" clause. The loader throws an error message, "Invalid query, must start with select".
Liam Any one know of a JEXL validator / tester? (other than trying to do stuff directly in java?)
Liam In the new provisioner, if I set the group name to be required, could I use the validation expression to prevent the allow and deny groups from being provisioned?
Erik our security team wants our 2FA-challenged group to be permanent-- you can get in, but you can't get out. So we want to create a "one-way" policy group,
Drew
In our Dev tier, we keep getting this error. I've tried to research it, and I find some references to others having had this, but I can't find causes or solutions.
Liam
I’m trying to set up a provisioner to AD, and I’m not sure what attributes are required.
Michael G
anyone interested in sharing experiences running Grouper with GKE in GCP? I'm looking to do it and hoping for advice/wisdom/perspective from those experienced doing so.
Bill K
Someone from Indiana University of Pennsylvania is interested in see a Grouper demo. Do we have any volunteers?
jasonrap
I need some help/ suggestions related to this article: https://spaces.at.internet2.edu/display/Grouper/Install+the+Grouper+v2.5+container+with+maturity+level+0+manually I am on step 29, none of the log files are outputting to my system.
Justin R
I am working with some email messages from Grouper and trying to insert a new line into the message from a rule. I’ve tried adding \n to the message, but it seems to be coming through as just a \n.
Liam
Anyone have suggestions as to a active directory attribute to use for the grouper group identifier other than gidNumber? We’re already using that for something else..
Liam
Googling for the grouper javadocs sends me here - https://software.internet2.edu/grouper/doc/2.5.x/grouper-parent/index.html - and I’m not seeing any documentation, just liinks to mostly blank pages. Am I in the right place?
Michael L
my first research task is to extract a list of "folders" from our grouper environment for Splunk purposes...i'm hoping this is a place that can help me with that
Next Grouper Call: Wed. March 3, 2021