Friday, September 20, 2013

Registry Services on the glass

glassAnother entry related to my current project, Jazz for Service Management. I am quite thrilled about having mixed jQuery and rdfQuery to interact with our OSLC server directly from a web-browser. Look’ma, only 2-tiers:

Wednesday, September 18, 2013

Renewing expired certificates for wsadmin invocations against WebSphere Application Server

We have a verification lab with about a dozen machines and on a periodic basis we hit a problem where wsadmin cannot connect to an instance of WebSphere Application Server due to a certificate expiration.

Note: This being a closed test environment, this kind of problem can be minimized or avoided altogether by creating the profile in Advanced mode through the Profile Management Tool and choosing longer expiration period than the default one-year period chosen when creating the profile using the manageprofiles command-line tool.

The connection error message returned by wsadmin will not be all that helpful because essentially it does not know why its connection request was refused, but the server logs ($profile/logs/server1/ServerOut.log or TextLog_<timestamp>.log depending on your troubleshooting settings) will contain a very telling message like this:

[9/18/13 10:22:53:664 EDT] 0000001c WSX509TrustMa E   CWPKI0022E: SSL HANDSHAKE FAILURE:  A signer with SubjectDN "CN=hostname, OU=hostNode01Cell, OU=hostNode01, O=IBM, C=US" was sent from target host:port "unknown:0".  The signer may need to be added to local trust store "$profile/AppSrv01/config/cells/hostNode01Cell/nodes/hostNode01/trust.p12" located in SSL configuration alias "NodeDefaultSSLSettings" loaded from SSL configuration file "security.xml".  The extended error message from the SSL handshake exception is: "PKIX path validation failed: The certificate expired at Wed Aug 21 21:54:27 EDT 2013; internal cause is: NotAfter: Wed Aug 21 21:
54:27 EDT 2013".

It took me a little while to figure out the keystore used by wsadmin, which turned out to be:


Listing the certificates in the file quickly showed the offending certificate (keep in mind that the default keystore password in WAS is WebAS) :

bash-2.04# keytool -list -storetype PKCS12 -keystore ./profiles/AppSrv01/etc/key.p12 –storepass <password> -v
Keystore type: PKCS12
Keystore provider: IBMJCE
Your keystore contains 1 entry
Alias name: default
Creation date: Aug 22, 2012
Entry type: keyEntry
Certificate chain length: 2
Owner: CN=hostname, OU=hostNode01Cell, OU=hostNode01, O=IBM, C=US
Issuer: CN=hostname, OU=Root Certificate, OU=hostNode01Cell, OU=hostNode01, O=IBM, C=US
Serial number: 2623bc78cd3b
Valid from: 8/21/12 9:54 PM until: 8/21/13 9:54 PM
Certificate fingerprints:
         MD5:  D2:FB:3F:8A:53:6F:19:B2:6C:77:AB:00:AC:40:EC:0B
         SHA1: 64:76:8F:7F:2A:0A:6A:F0:C9:21:86:FF:90:5B:C5:FA:FF:64:61:B4
Owner: CN=hostname, OU=Root Certificate, OU=hostNode01Cell, OU=hostNode01, O=IBM, C=US
Issuer: CN=hostname, OU=Root Certificate, OU=hostNode01Cell, OU=hostNode01, O=IBM, C=US
Serial number: 2622d2d16331
Valid from: 8/21/12 9:54 PM until: 8/18/27 9:54 PM
Certificate fingerprints:
         MD5:  C2:CB:AB:C3:3A:6C:D2:77:6A:87:DA:D7:21:2E:DC:E4
         SHA1: 0C:09:CD:37:96:F8:FE:91:18:92:5A:93:05:AB:15:D6:6D:36:A1:AA

One can (nay, should) use the keytool command-line to recreate the certificate above, but this being a closed test environment where we do not test SSL functionality, and since we did not have any new certificates added to either wsadmin or server keystores since we installed that test system, I did the unthinkable, that is, overwrite the wsadmin keystore with the server keystore.

cp $profile/config/cells/hostNode01Cell/nodes/hostNode01/key.p12 $profile/etc/key.p12

Note: The last step was a very specific decision for a test environment running WebSphere Application Server. **DO NOT** take the last step on a production environment or where company policies do not allow for it, and only after carefully examining the two keystores to ensure all their fields (sans expiration date) are exact matches.