Upgrading to Grouper 2.2 from Grouper 2.1
Using the Grouper Upgrader can simplify your upgrade process. Here is a movie demonstrating the Grouper upgrader. The upgrader can upgrade an installed env of the API, UI, WS, client, PSP, etc. If you dont have a build script to manage multiple envs, you might want to use the upgrader.
These instructions describe how you can upgrade to Grouper 2.2 from 2.1. Note that if you are upgrading from 1.6 or 2.0, there are special notes below on what to do differently.
2014/09/01: for 2.2.1+ For the UI, edit the tomcat server.xml <Connectors to have uri encoding of utf8: <Connector URIEncoding="UTF-8"
2015/01/18: If you have installed UI patches before this date, you should edit the grouperPatchStatus.properties file (in WEB-INF). Change two instances of grouper_v2_2_1_ui_patch_3 to grouper_v2_2_1_api_patch_2. These are the same patch and it belongs in the API not UI.
2015/09/30: Make sure to run the grouperInstaller of the version of grouper you are upgrading from to revert all patches before upgrading. Note, If reverting patches doesn’t work, and you skipped that part in an upgrade, then before new patches are applied, look through the WEB-INF/classes dir for classfiles, and if you have never overridden or coded your own, then delete all classfiles in the subdirs. If you have, then keep yours, and delete the others.
Important Changes in Grouper 2.2 that impact the upgrade
Configuration Files: In order to make Grouper more easily deployable across environments, and more easily upgradable, Grouper now has configuration overlay files, and the ability to use expression language in config file entries. It is highly recommended that as you upgrade to Grouper 2.2, you restructure your configuration files to take advantage of the configuration overlay files.
Here is how it works:
- There is default configuration file (called the base file), and there is an override (overlay) file that includes only the changes from the default. Several, but not all, of Grouper's configuration files now use this structure.
- In the past, the grouper.properties file would contain all the default properties along with your changes.
- Starting with Grouper 2.2, the grouper.properties file should only contain your changes from the base file. Note: "grouper.properties" is used as an example file name here, there are several other files such as grouper-ws.properties, etc.)
- If you don't know what your changes were from the base configuration, you should diff the file with the example file from that release (i.e. diff grouper.properties grouper.example.properties).
- The results of the diff should go into the override file.
- The base file should be used as provided.
- See Grouper configuration overlay for more details and which configuration files use this new method.
- The sources.xml file does not use the overlay method. Also, the sources.xml file from 2.1 is compatible with 2.2.
Legacy Style Attributes: The legacy style attributes and group types are no longer part of Grouper. Those legacy APIs still work though; they just use the newer attribute framework. This upgrade process includes migrating those legacy attributes and group types to the newer attribute framework. Read more about the legacy attribute and group type migration.
- If you are unsure about the legacy group types that you have, you can look in the grouper_types table. The group types "base", "naming", and "attributeDef" were all internal and no longer apply. If you have custom code that tries to find those group types now, you will receive an exception. The rest will be migrated including any built-in group types that you may be using (e.g. addIncludeExclude, requireInGroups, grouperLoader). Also note that the legacy group types are visible in the admin UI when viewing/updating a group.
- If you are unsure about the legacy attributes that you have, you can look in the grouper_fields table for rows where the type column is set to "attribute". All of these attributes will be migrated to the newer attribute framework.
New Privileges: New privileges have been added to determine who has read and update access to attributes based on what the attribute is assigned to. The new privileges are: groupAttrRead, groupAttrUpdate, stemAttrRead, stemAttrUpdate, attrDefAttrRead, and attrDefAttrUpdate. These privileges are not set during the upgrade, so if your users need read or update access to attributes and they are not admins of those objects, then the appropriate privilege would need to be assigned. This applies to attributes originally created using the newer attribute framework as well as attributes using the old attribute framework that would be migrated during this upgrade. See the section of the Attribute Framework page titled New Privileges for Attributes in Grouper 2.2+
GrouperAll (aka EveryEntity): Starting from 2.2.1, GrouperAll isn't allowed to be granted ADMIN, UPDATE, or GROUP_ATTR_UPDATE privileges on groups. It is also not allowed to become a member of a group. If you are upgrading from 2.1 (or before) directly to 2.2.1+ (thus skipping 2.2.0), then the upgrade steps below will remove these privileges and memberships. If you are upgrading from 2.2.0 to 2.2.1+, then run "gsh misc/postGrouper2_2_1Upgrade.gsh". The upgrader also takes care of this.
Other items before upgrading
- You may want to have your DBAs make sure you are not close to running out of tablespace. In general, it may be useful to have your DBAs available when you upgrade.
- If you have views that other systems use, you could replace them as tables before beginning.
- If you have other systems using Grouper, you could temporarily disable them.
- If you have a large number of groups/folders, it would probably be a good idea to add temporary indexes on grouper_change_log_entry_temp.string01 and grouper_change_log_entry_temp.string02 to help speed up Step 9 below. You can drop the indexes after the upgrade. (The index creation will probably later be added automatically during the upgrade but it does not do that right now.)
The upgrade adds new unique indexes to the point in time audit tables. The index creations may fail if you have duplicate rows, which could have happened if you ever had multiple instances of the Grouper daemon running at the same time. Check to make sure that the following queries don't return anything before starting the upgrade. If they do, then the duplicate rows would need to be cleaned up first.
- If you are upgrading from 1.6 or 2.0, take a look at item #1 in the 2.1 upgrade instructions.
- You should get v2.2 versions of the Grouper API, Grouper UI, Grouper WS, Grouper Daemon, etc. from the Grouper Downloads page. You will need to merge configuration files and JARs. See the v2.2 change log for more information. Also keep in mind that some of the configuration files are now handled differently via configuration overlay. The rest of this document focuses on upgrading the database.
- First you may want to analyze your tables to help speed up the upgrade. Analyze your tables.
Stop the Grouper Daemon. Once you prevent users from making updates to your Grouper instance, run the changeLogTempToChangeLog daemon to clear out the temp changelog using your existing v2.1 API. Here's an example using GSH.
- Before performing any upgrade steps, export your Grouper registry. Options include performing a database backup (recommended) or using the XML Export utility in Grouper (not recommended since certain features may not get exported).
Using the 2.2 API, perform a registry check using GSH to create an SQL file that will contain the DDL to update your database. To do this, run: gsh -registry -check Note you may need to increase memory. For instance..
- In this example above, an SQL script called /opt/grouper/ddlScripts/grouperDdl_20140523_13_19_30_162.sql was created.
- Postgres only - If using postgres, you should see foreign keys being dropped at the top of the script. If not, try setting the ddlutils.schema grouper.properties setting and run again. If you still don't see foreign keys being dropped at the top of the script, manually drop all foreign keys before running the script.
- Postgres and hsql only - You should backup any non grouper views that depend on Grouper views, run the grouper script (which deletes those views due to drop view cascade), and then you should recreate those non grouper views.
Run the SQL script. To do this, run: gsh -registry -runsqlfile /path/to/sql/file.sql For instance..
Note that if one of the SQL statements in the script fails, the process will abort leaving the rest of the SQL statements from executing. If this happens, in most cases, you can't just re-run the full script since re-executing some of the DDL changes that previously succeeded would fail now (e.g. dropping a view or constraint that was previously dropped successfully.) You could edit the script to remove the statements that previously succeeded in order to re-execute the statement that failed and the ones after it. Or you can run the previous step again to generate a new SQL script.
- If you are upgrading from 1.6, take a look at item #14 in the 2.1 upgrade instructions.
Now that the DDL updates have been made, there are a few additional GSH commands that need to be run. To do this, run: gsh ../misc/postGrouper2_2Upgrade.gsh (The gsh script is in the "misc" directory.) Note you should check the output to make sure no errors are thrown. If you see an error, it is safe to re-run. For instance..
Note that for the 7th step above (Add group sets for new privileges to point in time (groupAttrRead, groupAttrUpdate, stemAttrRead, etc)), if it will be adding more than 100,000 group sets, then it will process them 100,000 at a time. If you would like to see the progress as it is processing each set of 100K, tail the grouper_error.log file.
Run: gsh ../misc/postGrouper2_2_1Upgrade.gsh (The gsh script is in the "misc" directory.)
- Analyze your tables. (Again to avoid any performance issues.)
- If you are upgrading from 1.6 or 2.0, take a look at item #16 in the 2.1 upgrade instructions.
- Start the Grouper Daemon and all other Grouper components (UI/WS).
- The legacy attributes were backed up in separate tables. After verifying that everything is okay, you can drop those backed up tables by setting ddlutils.dropLegacyAttributes = true in grouper.properties, running gsh -registry -deep and then executing the SQL file.