I am chasing down an issue with do-send-email-from-template. This is something that was working, post upgrade, but now has quit.

The error I get is this:

DirXML Log Event -------------------
Driver: \TREENAME\Services\DriverSetName\Company_AD_Driver _Stage
Channel: Subscriber
Object: \TREENAME\CompanyName\TestUser002
Status: Error
Message: Code(-9195) Error in vnd.nds.stream://TREENAME/Services/DriverSetName/Company_AD_Driver_Stage/itp-SendNewUserInfo#XmlData:336 : Couldn't send email: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderE xception: unable to find valid certification path to requested target
[05/24/19 09:22:29.576]:MF_AD_STAGE ST: Query from policy
[05/24/19 09:22:29.576]:MF_AD_STAGE ST:

I found a related (somewhat) TID (forgot the number) that said to add the smtp server's cert to the cacert keystore. Did that. The Cert signers were already in the keystore. No joy.

I started checking access using nc, dig, host, and nslookup, just to verify settings and found an interesting problem. It seems this server has a mismatch in where it is getting it's library files.
This is SLES 12 SP4. eDir 9.1.3, IDM is 4.7.2, AD Driver is 4.1.1.0.

This environment has 3 virtually identical servers. I tried doing the send-email on another server, and it works fine. Digging deeper, I used the linux ldd command to trace some libraries, and found this.

Server 1 is the working server, Server 2 is the one that has the email problem. Server 1 did NOT get the smtp server cert added to cacerts, yet works fine.

Using ldd on openssl, here are the differences...

server1:~ # ldd -v /usr/bin/openssl
linux-vdso.so.1 (0x00007ffdd7f36000)
libssl.so.1.0.0 => /lib64/libssl.so.1.0.0 (0x00007f48031de000)
libcrypto.so.1.0.0 => /lib64/libcrypto.so.1.0.0 (0x00007f4802d76000)
libc.so.6 => /lib64/libc.so.6 (0x00007f48029d1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f48027cd000)
libz.so.1 => /lib64/libz.so.1 (0x00007f48025b6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f480344c000)

Version information:
/usr/bin/openssl:
libc.so.6 (GLIBC_2.15) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libssl.so.1.0.0 (OPENSSL_1.0.0) => /lib64/libssl.so.1.0.0
libcrypto.so.1.0.0 (OPENSSL_1.0.0) => /lib64/libcrypto.so.1.0.0
/lib64/libssl.so.1.0.0:
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libcrypto.so.1.0.0 (OPENSSL_1.0.0) => /lib64/libcrypto.so.1.0.0
/lib64/libcrypto.so.1.0.0:
libdl.so.2 (GLIBC_2.2.5) => /lib64/libdl.so.2
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.7) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
/lib64/libc.so.6:
ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
/lib64/libdl.so.2:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/lib64/libz.so.1:
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6

server2:~ # ldd -v /usr/bin/openssl
/usr/bin/openssl: /opt/novell/lib64/libssl.so.1.0.0: no version information available (required by /usr/bin/openssl)
/usr/bin/openssl: /opt/novell/lib64/libcrypto.so.1.0.0: no version information available (required by /usr/bin/openssl)
linux-vdso.so.1 (0x00007ffd638df000)
libssl.so.1.0.0 => /opt/novell/lib64/libssl.so.1.0.0 (0x00007f6486dcb000)
libcrypto.so.1.0.0 => /opt/novell/lib64/libcrypto.so.1.0.0 (0x00007f64869bd000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6486618000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6486414000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6486d2e000)

Version information:
/usr/bin/openssl:
libc.so.6 (GLIBC_2.15) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.14) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libssl.so.1.0.0 (OPENSSL_1.0.0) => not found
libcrypto.so.1.0.0 (OPENSSL_1.0.0) => not found
/opt/novell/lib64/libssl.so.1.0.0:
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/opt/novell/lib64/libcrypto.so.1.0.0:
libdl.so.2 (GLIBC_2.2.5) => /lib64/libdl.so.2
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/lib64/libc.so.6:
ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
/lib64/libdl.so.2:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_PRIVATE) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6

There are missing libraries, and for Server2, it seems to be getting some libraries from /opt/novell/lib64, and others from /lib64.

/lib64 has newer (by date/time stamp) files than /opt/novell/lib64

There were similar results using ldd on dig, host, and nslookup. It appears that something may have updated/patched Server2 via SUSE, and changed something, but what?

An echo $LD_LIBRARY_PATH shows nothing on both, and their $PATH statements are identical. Both servers have iManager 3.0.3 installed, but Server2 has started throwing this: Unable to create AdminNamespace. java.lang.NoClassDefFoundError: novell/jclient/JCException where it worked fine previously.

This is our testing environment, and we need this answered before we move Production to IDM 4.7.2. Prod runs on SLES 11 SP3, and eDir 8.8.8.8, so this is a multi-step upgrade. Not very easy to roll back.

Thoughts or ideas?