Hacking on ThinPro: Fixing EAP-TLS

I ran into a problem awhile ago which was causing ThinPro to strip an issuing CA out of a certificate chain. My investigation started when I noticed that wireless authentication for my test device was failing in certain areas. Naturally, I wanted to see what was going on with the wireless NIC, so I ran grep -i wlan0 /var/log/syslog to see if I could catch why it was so unhappy. I found the following information:

wlan0: CTRL-EVENT-EAP-TLS-CERT-ERROR reason=1 depth=0 subject='/C=US/ST=NY/L=HomeBase/O=NewbAdmin/OU=DummyOU/CN=DummyDev' err='unable to get local issuer certificate'
...
SSL: SSL3 alert: write (local SSL3 detected an error):fatal:unknown CA
EAP: Status notification: local TLS alert (param=unknown CA)
...
OpenSSL: openssl_handshake - SSL_connect error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

Seeing the unknown CA alert, I thought maybe I had neglected to add a certificate, so I checked ThinPro's certificate control panel, but I didn't see anything missing. Next, I sent a SCEP enrollment task from HP Device Manager (HPDM) to see if ThinPro was breaking something during the enrollment process. Here's a screenshot of the chain before SCEP enrollment:

The purpose of a PKI certificate is to be public, but I used dummy certificates here anyway because why not.

As you can see, there are clearly two certificates in this chain, but after running the SCEP enrollment task, I found this:

Yup, that would certainly do it. Something happened during the SCEP enrollment process that chopped out the second certificate. Also, RUDE!

Now that I could reproduce the problem on demand, I wanted to know when the OS started behaving this way, so I started rolling back updates. I started with SP8 and confirmed the issue was still present. The problem was also still present in SP7. It wasn't until I rolled back to SP2 that I was able to confirm that certificate chain processing was behaving correctly. So what changed?

Unfortunately, there wasn't anything helpful in the service pack release notes, so I had to manually update the OS and check the update log (hurray....) The first thing that caught my eye was that the hptc-cert-mgr package had been updated.

I found the /lib/hptc-cert-mgr/folder and took a look at the contents.

Hey, look at that! There's a script in there called certmgr-apply. That sure sounds promising. I compared the same script from SP2 and SP9. After confirming that the script had indeed been modified between versions, I reinstalled the copy from SP2 and re-ran the SCEP enrollment task. To my great relief, ThinPro was no longer breaking the chain during enrollment.

My experience with vendor support has been that more information = more better, so I decided to dig a little deeper before reporting the bug to HP. I compared the two copies of the certmgr-apply script and saw that there were only a few changes. I'll spare you the boring trip through most of the code, but my particular problem stems from this block here:

else
    openssl x509 -in $CA_CERT_FILE -inform PEM -out $DEST_FILE.crt -outform PEM
    if [ "$?" = "0" ]; then
        echo "+${LOCAL_CERT_STORE}${DEST_FILE}.crt" >> ${CERTMGR_CHANGE_FILE}
        rm $CA_CERT_FILE
    fi
fi

This part of the script basically extracts the first certificate in the chain, outputs it to a new file, and then deletes the original chain. I'm not entirely sure what ultimate purpose this serves, but there is a very similar block of code with the same openssl command in both versions of the script, so I'll assume that it's at least trying to do something important.

The critical difference for me is actually closer to the top where some variables get declared. Here's the SP2 copy:

SUPPORT_SUFFIX=".crt .CRT"
UN_SUPPORT_SUFFIX=".pem .PEM"
TO_CONVERT_SUFFIX=".der .DER .p7b .P7B .p7c .P7C .cer .CER"

And here's the SP7 copy:

SUPPORT_SUFFIX=""
UN_SUPPORT_SUFFIX=".pem .PEM"
TO_CONVERT_SUFFIX=".crt .CRT .der .DER .p7b .P7B .p7c .P7C .cer .CER"

My walk through the script revealed that the removal of .crt from the SUPPORT_SUFFIX variable causes the certificate chain to be processed by the broken openssl command. The openssl command causes the issuing CA to be stripped out of the chain, and that results in ThinPro refusing to connect to the wireless network.

I hope someone out there on them internets finds this useful or at least mildly educational. Navigating a problem like this is rarely as quick or straightforward as it seems in the write-up, so remember to be patient and take it one step at a time.

Thanks for reading!

CCNA Prep: Applying ACLs in the Right Direction

CCNA Prep: Proxy ARP

CCNA Prep: Misinformation in Boson NetSim Labs

CCNA Prep: Setting Up Cisco Modeling Labs on Proxmox

Hacking on ThinPro: Enabling SSH

Hacking on ThinPro: Preventing User Interruptions

Hacking on ThinPro: Sneaky Workarounds

Hacking on ThinPro: Extension Properties

Hacking on ThinPro: Fixing EAP-TLS

Learning PowerShell: A Bash Fan's Experience

Learning PowerShell: Using PSComputerName

Learning PowerShell: Removing Unknown Profiles

Learning PowerShell: Finding Users in Folder Permissions

Learning PowerShell: Logging Pings

Hacky Nonsense: Bookstack Load Balancer

Hacking on ThinPro: Fixing EAP-TLS