At the November TAC F2F, we discussed having a matrix of best practices by which to evaluate registered sites to help set expectations and create peer pressure. This is a preliminary set of suggested criteria.
The Best Practice Matrix might also be thought of as the "Community Wall of (Fame | Shame)." Emphasizing "Fame" or "Shame" would be a conscious decision on our part. It seems both approaches would be appropriate and effective in practice. For example, the IDs of entities requesting the unsupported WAYF might be published on the "Wall of Shame" while a list of those entities supporting SAML V2.0 would be published on the "Wall of Fame."
Policy
...
- POP Available
- Appropriate Contacts
- Security Incident Contact Registered
- Does this also imply adherence to the recommended incident response process?
- Process for facilitating attribute release to SPs
- Release of basic attributes to SPs in some TBD automated fashion (with or without consent)
Deployment Practices
Technical Basics
- Maintaining Unexpired Certificates
- Accessing the (unsupported) InCommon WAYF
- Deploying an Unsupported Version of Shibboleth / OS / Web Server / etc.
- Expired Certificates in Metadata
- SAML 2.0 Support
- IdPs with TLS-protected HTTP-Redirect SSO
- SPs that support SAML 2.0 should indicate so in metadata
- SPs with TLS-protected HTTP-POST ACS and an encryption key
- SAML 1.1 Support
- SPs with TLS-protected HTTP-POST ACS
Operational Maturity
- Maintaining Supported Software
- Software Compliance with Metadata IOP
- Federation a "First Order" UI
- Discovery
- Choices offered should result in an "acceptable" experience
- Error Handling
- Look and Feel
- Useful Contacts
Maximizing the Federation
- Documented Attribute Release Process
- Support for SAML 2.0 "persistent NameIDs " NameID or eduPersonTargetedID
- Perhaps support for other attributes are worth noting?
- Full saml2int conformance
- Consent-based support for particular attributes (i.e., no admin involvement needed)
- Release of "basic" attributes w/o admin involvement (via consent or otherwise)
Parked Items
- Keys of less than a certain age
- We should consider what, if any, age is actually "too old"
- Appropriate error pages
- Perhaps subjective, but I'd start with having actual contact info for users and a reasonable indication of what to do, maybe not using the Shibboleth logo?
Implementation Support
- Full saml2int conformance
- InCommon Implementation Profile conformance
- Could call out Metadata IOP as a subset, but my guess is few products would support that without the rest
- Could identify "exceptions to conformance" to highlight specific missing capabilities or could break profile into separate features in the matrix
...