Troubleshooting [dynes:IN PROGRESS]
- General (CERT related) Caveats
When setting up the Remote Link in oess the name of the link needs to be the same as the link= string at the other end. The urns in the topology file for each end of the link must match exactly. Here are two ends of a link taken from the ION topology information at https://ndb7.net.internet2.edu/TopologyViewer/?domain=dynes.internet2.edu&ts_instance=http%3A%2F%2Fdcn-ts.internet2.edu%3A8012%2FperfSONAR_PS%2Fservices%2Ftopology. Note that the I2 NOC must set this information up for each remote site. So, whatever you tell the NOC to put in the link= field of the first item below is what you need to use at your end. urn:ogf:network:domain=dynes.internet2.edu:node=sw-i2lab.dynes.internet2.edu:port=Te+0/0:link=internet2 urn:ogf:network:domain=ion.internet2.edu:node=rtr.chic:port=xe-7/1/2:link=lab-idc In oess when setting up a Remote Link the box asks for Name and Remote URN. In the above example, Name should be "internet2" whatever follows link= in the first item above (what you told the NOC to put there). The second line is the urn of the link to you from the other end's perspective. This is what you should put in the Remote URN field when setting up the remote link in oess. CERTS CERTS are stored in /etc/pki/tls/certs (key in /etc/pki/tls/private). The CERT for the site must also be inserted into /etc/oscars/keystores/oscars.jks and /etc/oscars/keystores/localhost.jks. ./idc-importkeypair -s /etc/oscars/keystores/oscars.jks -a mykey -k /path to/yourhost.key -c /path to/yourhost.crt -p changeit -O ---- copy oscars.jks to localhost.jks ----The keystores must also contain a CERT for localhost. See below for instructions ---- on generating that CERT. ---- Whatever you call the CERT and KEY you install in /etc/pki/tls you must also ---- update /etc/httpd/conf.d/ssl.conf SSLCertificateFile /etc/pki/tls/certs/your.host.crt SSLCertificateKeyFile /etc/pki/tls/private/your.host.key ---- Also these paths must be placed in /etc/oess/database.xml cert="/etc/pki/tls/certs/your.host.crt" key="/etc/pki/tls/private/your.host.key" ---- The owner and issuer fields for the local site CERT need to be placed in the local IDC ---- user in OSCARS. These also need to be placed in the IDC user at the peer host. See ---- below regarding changing the IDC peer. ---- In case all you have is the old (.5) OSCARS.jks do the following to get the cert and key: keytool -v -importkeystore -srckeystore OSCARS.jks -srcstorepass changeit -srckeypass password -srcalias rutgers_idc -destkeystore temp.p12 -destalias mykey -deststorepass changeit -destkeypass changeit -deststoretype PKCS12 ---- Now the cert and key are in a .pem file, do this: openssl x509 -in temp.pem > yourhost.crt openssl rsa -in temp.pem -out yourhost.key ---- Copy the cert and key to /etc/pki/tls as mentioned above. Don't forget to update ssl.conf!
- More CERT related information...
-- Suffixes: .cer, .crt CER is an X.509 certificate in binary form, DER encoded. CRT is a binary X.509 certificate, encapsulated in text (base-64) encoding. It is not the same encoding but they are otherwise interchangeable. -- A .cer file containing PKCS#7-encoded data looks like this: -----BEGIN PKCS7----- MIIW4gYJKoZIhvcNAQcCoIIW0zCCFs8CAQExADALBgkqhkiG9w0BBwGggha1MIIH ... POI9n9cd2cNgQ4xYDiKWL2KjLB+6rQXvqzJ4h6BUcxm1XAX5Uj5tLUUL9wqT6u0G +bKhADEA -----END PKCS7----- -- To convert a pkcs7 .cer to .crt openssl pkcs7 -print_certs -in certificate.cer -out certificate.crt -- A PEM certificate data looks like this: -----BEGIN CERTIFICATE----- MIIHNjCCBh6gAwIBAgIQAlBxtqKazsxUSR9QdWWxaDANBgkqhkiG9w0BAQUFADBm ... nv72c/OV4nlyrvBLPoaS5JFUJvFUG8RfAEY= -----END CERTIFICATE----- If the .cer is in PEM format it's the same as a .crt.
- To install/replace the CERT on the IDC
The steps to import the cert and key and update the user in the OSCARS webUI. This works for both .pem and .crt files. -- If cert is in a .pem file do this: openssl x509 -in yourcert.pem > yourhost.crt openssl rsa -in yourcert.pem -out yourhost.key -- Import the cert and key to the oscars.jks and localhost.jks to overwrite the previous entry cd /opt/oscars/tools/bin/ ./idc-importkeypair -s /etc/oscars/keystores/oscars.jks -a mykey -k /path to/yourhost.key -c /path to/yourhost.crt -p changeit -O ./idc-importkeypair -s /etc/oscars/keystores/localhost.jks -a mykey -k /path to/yourhost.key -c /path to/yourhost.crt -p changeit -O -- (Optional) To check the owner, issuer, and validation date stored in oscars.jks -- See below for instructions to extract the cert and key keytool -list -v -alias mykey -keystore oscars.jks |egrep "Owner|Issuer|Valid" -- Retrive the updated subject and issuer and update the Local IDC Account (yoursite-idc) in the OSCARS webUI: https://yourhost.edu:8443/OSCARS/ keytool -printcert -file /path to/yourhost.crt|egrep "Owner:|Issuer:" Owner: CN=<whatever is there>, OU=Services, DC=doegrids, DC=org Issuer: CN=DOEGrids CA 1, OU=Certificate Authorities, DC=DOEGrids, DC=org -- Copy the new cert and key to the proper places -- Make sure permissions are 644 -- Also make sure that only one cert is inthe cert file -- and the CN in the cert must match the host name cp /path to/yourhost.crt /etc/pki/tls/certs cp /path to/yourhost.key /etc/pki/tls/private -- To check that the keys are the same openssl rsa -in /etc/pki/tls/private/yourhost.key openssl x509 -in /etc/pki/tls/certs/yourhost.crt -- Update the new cert and key locations in the relevant places (if they've changed) -- If you make a mistake in database.xml you'll get "Could not send Message" in the OSCARS UI /etc/oess/database.xml /etc/httpd/conf.d/ssl.conf -- Restart OSCARS service oscars restart -- Restart HTTPD service httpd restart -- Restart OESS (may not be necessary) service oess restart
- To generate a new self-signed CERT
-- -nodes means no PEM pass phrase is used -- -days 1825 = 5 years use what makes sense openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 1825 -nodes -- Enter relevant information at the prompts -- Common Name (eg, your name or your server's hostname) must match the actual server name -- Copy resulting cert.pem and key.pem to /etc/pki/tls -- If you don't overwrite what was already in /etc/pki/tls -- then you must update /etc/httpd/conf.d/ssl.conf cp cert.pem /etc/pki/tls/certs/your.host.crt cp key.pem /etc/pki/tls/private/your.host.key -- Now update the cert used by OSCARS cd /etc/oscars/keystores/ keytool -delete -alias mykey -keystore oscars.jks -storepass changeit /opt/oscars/tools/bin/idc-importkeypair -s oscars.jks -a mykey -c /etc/pki/tls/certs/your.host.crt -k /etc/pki/tls/private/your.host.key -p changeit -O -- Hit return when prompted Enter PEM pass phrase: -- IMPORTANT!! oscars.jks and localhost.jks need a cert corresponding to the actual local host name but also one for localhost. openssl req -x509 -newkey rsa:2048 -keyout localhostkey.pem -out localhostcert.pem -days 1825 -nodes Generating a 2048 bit RSA private key ................+++ ..........................................................................................+++ writing new private key to 'localhost.pem' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:US State or Province Name (full name) []:CA Locality Name (eg, city) [Default City]:. <- Note period entered here to blank out default Organization Name (eg, company) [Default Company Ltd]:OSCARS Organizational Unit Name (eg, section) []:OSCARS DEFAULT Common Name (eg, your name or your server's hostname) []:localhost Email Address []: -- Verify that the stupid CERTs were inserted correctly keytool -list -alias mykey -v -keystore /etc/oscars/keystores/oscars.jks -storepass changeit keytool -printcert -file /etc/pki/tls/certs/your.host.crt -- Update localhost.jks cp oscars.jks localhost.jks -- Restart everything service httpd restart service oscars restart service oess restart
- To install/replace the CERT on the FDT server
The cert and the key are stored under: /home/dynes/FDTAgent/conf/OSCARS.jks To import a new certificate and the key: 1) Go to /home/dynes/FDTAgent/bin 2) Import the certs using the `dynesfdt cert` command: ./dynesfdt cert -import -key </path/to/private/key> -cert </path/to/cert> The certs will be imported under a new file (in case the user wants to backup the previous one): /home/dynes/FDTAgent/conf/OSCARS.jks.NEW This file must be copied over: /home/dynes/FDTAgent/conf/OSCARS.jks 3) To list the certificate currently installed in the OSCARS.jks file: ./dynesfdt cert should print the certificate and the validity. - Virtual Machine Issues
The IDC platform runs one or more virtual machines. Each VM typically runs an instance of OSCARS along with software that controls the DYNES switch. Running task is /usr/libexec/qemu-kvm. It might show up in top as qemu-kvm. service libvirtd restart if you need to kill and restart the task. Some useful VM related commands: Command Function virsh The virsh program is the main interface for managing virsh guest domains. See man virsh for details. virsh list --all List all the VMs. virsh start <domain> Start the domain (either by name or number as given by virsh list --all virsh shutdown <domain> Shutdown the domain virsh console <domain> Connect to the VM's console. Ctrl-] to exit. virsh edit <domain> Edit the domain's XML. virsh dominfo <domain> Print out info about the domain. yum install libguestfs-tools will install a set of useful VM related tools. - Circuits that don’t terminate
- Instructions on how to interact with the OESS database
- Error in signature with X509Token
log into your OSCARS UI: https:idc.net.<site>.edu:8443/OSCARS/ lookup the reservation examine the error message. - Can't access both the IDC and FDT iDRAC using Firefox. This is due to all Dell iDRACs using the same cert serial number. Firefox (for some reason) doesn't allow this to be ignored and continue. The workaround is to delete the certs from the first iDRAC accessed or use another browser. Details on removing the certs can be found here.
- Mistakenly "Decommissioned" the switch
# mysql -u root -p Enter password: <enter mysql root password> mysql> use oess; mysql> select * from node; <- Your switch will likely be node=1 Decoming the device sets an end_epoch for it in the node_instantiation table. You must login to mysql as root. mysql> select * from node_instantiation; ----------------------------------------------------------------------------+ node_id end_epoch start_epoch management_addr_ipv4 admin_state dpid ----------------------------------------------------------------------------+ 1 1376100241 1374712867 2170417085 active 281483173139793 ----------------------------------------------------------------------------+ 1 row in set (0.00 sec) Set the end_epoch to -1 and it'll be active. mysql> update node_instantiation set end_epoch =-1 where node_id =1; Query OK, 1 row affected (0.04 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> select * from node_instantiation; ---------------------------------------------------------------------------+ node_id end_epoch start_epoch management_addr_ipv4 admin_state dpid ---------------------------------------------------------------------------+ 1 -1 1374712867 2170417085 active 281483173139793 ---------------------------------------------------------------------------+ 1 row in set (0.00 sec)
- Reported by MAX
Problem: -- Decommissioned the node -- Lost the node and could not rediscover -- Deleted oess database -- Reran oess_setup.pl to restore database to clean state -- Discovered and re-added the node and all interfaces -- re-add remote link and click "Submit Topology" This didn't re-submit the topology I believe the correct steps are: -- Go 'admin' page -- Click 'Workgroups' then choose 'OSCARS IDC' -- Add new interfaces in "Allowed Edge Ports" -- Click "Remote Links" then click "Submit Topology"
- Change IDC peer link
-- NOTE! Be sure to specify https in the url of the IDC! -- First display the current IDC peer # /opt/oscars/lookup/bin/oscars-idclist -- This will list Location: under Protocols: and domain= under Relationships: -- Example output for Internet2 as the current peer -- ID: 1 Type: IDC Expiration: NEVER Protocols: Type: http://oscars.es.net/OSCARS/06 Location: https://ion.net.internet2.edu:9001/OSCARS Relationships: [controls] urn:ogf:network:domain=ion.internet2.edu ID: 2 Type: IDC Expiration: NEVER Protocols: Type: http://oscars.es.net/OSCARS/06 Location: https://ion.net.internet2.edu:9001/OSCARS Relationships: [controls] urn:ogf:network:domain=ion.internet2.edu -- -- Now delete the current peer specifying -d and -l from the above output for your site -- # /opt/oscars/lookup/bin/oscars-idcdel ./oscars-idcdel A domain must me specified Option Description ------ ----------- -c, --context context in which to run the client -d, --domain required. the domain with the IDC to delete -h, --help prints this help screen -u, --url the URL of the OSCARS lookup module to contact -- -- Now add the new peer -- /opt/oscars/lookup/bin/oscars-idcadd A domain must me specified Option Description ------ ----------- -c, --context context in which to run the client -d, --domain required. the domain to add -h, --help prints this help screen -l, --location required. the URL of the IDC -p, --protocol the protocol spoken by the IDC. May be OSCARS5, OSCARS6, or a namespace URL. Defaults to OSCARS6. -u, --url the URL of the OSCARS lookup module to contact -- -- Sample (use this to add Internet2's ION as a peer) -- /opt/oscars/lookup/bin/oscars-idcadd -d ion.internet2.edu -l https://ion.net.internet2.edu:9001/OSCARS Note: The new peer must also do oscars-idcadd or equivalent. If Internet2 is being added as a peer please contact the Internet2 NOC giving your location and domain information along with your X.509 certificate Subject and Issuer. This must be a signed certificate if peering with Internet2. -- -- To modify a current peer's information. -- # /opt/oscars/lookup/bin/oscars-idcmod A domain must me specified Option Description ------ ----------- -c, --context context in which to run the client -d, --domain required. the domain with the IDC to modify -h, --help prints this help screen -l, --location the URL of the IDC -p, --protocol the protocol spoken by the IDC. May be OSCARS5, OSCARS6, or a namespace URL. Defaults to OSCARS6. -u, --url the URL of the OSCARS lookup module to contact - Errors due to OSCARS not being able to access localhost.jks or oscars.jks
-- Make sure both files are owned by the id oscars # ll /etc/oscars/keystores/ total 32 -rw------- 1 oscars oscars 7083 Feb 4 19:30 localhost.jks -rw------- 1 oscars oscars 7083 Jan 17 16:48 oscars.jks
- Extract cert and key from oscars.jks
# Extract cert from oscars.jks keytool -exportcert -alias mykey -keystore oscars.jks -file oscars.crt # Extract the private key from oscars.jks keytool -v -importkeystore -srckeystore oscars.jks -srcalias mykey -destkeystore oscars.key.p12 -destalias mykey -deststoretype PKCS12 openssl pkcs12 -in oscars.key.p12 -out oscars.pem openssl rsa -in oscars.pem -out oscars.key # Result is oscars.crt and oscars.key both of which can be moved to /etc/pki/tls certs or private # You may need to service restart httpd
- Missing required permissions manifest attribute in main jar
This might occur when trying to run the java iDRAC Virtual Console Client. (Click Details in the Application Error alert.) Solution: Start the Java Console (in System Preferences) Click Security tab Add site to Exception list, e.g. http://idrac.site.edu:443 Set Security Level to Medium (remember to set it to High afterwards!)
- Set up remote dynes host to allow X11 interaction
---- Run XQuartz or equivalent on your base workstation (may need to xhost+ in the xterm window) ---- On remote host change X11Forwarding to yes in /etc/ssh/sshd.config service restart sshd ---- Make sure xauth is available. If not: yum install xauth ---- Might want to add an innocuous X app yum install xclock ---- Back on your base workstation use -Y on ssh ssh -Y dynes@idc.yourhost.edu ---- This should result in the following message: /usr/bin/xauth: creating new authority file /home/dynes/.Xauthority ---- To allow root to also run X11 apps: cd cp ~dynes/.Xauthority .