“Nothing is certain but death & taxes”. And certificate expiry.
I’ve dodged the first one so far this year, but the other two have made up for it – and even more so with certs, as I’ve dealt with several customers where their root or intermediate certs have all expired. And yes, if you were watching, there was that unfortunate incident back in June with the greiginsydney website. I don’t want to talk about it.
To make my life – and hopefully yours – that little bit easier for the next time this comes up I thought I’d document a check-list of places in your Lync/SfB deployment you might find certificates lurking just waiting to expire. Some of the below is only relevant if you have root or intermeds expiring, so bear that in mind as you’re checking them off.
- Front-End servers.
- Edge servers. Don’t overlook that this one has both internal (private) and public certs.
- Mediation servers.
- Director servers.
- Have you deployed any other Lync/SfB server roles? PChat?
- Check the topology. Look for:
- Trusted Application servers.
- PSTN Gateways / SBCs running TLS.
- “Get-ExchangeCertificate | ft thumbprint,services,notafter -auto”
- If you’re sending IMs via OWA the old certificate’s thumbprint has been hard-coded in “web.config” & will need updating.
- KEMP / NetScaler / F5 / generic reverse proxy / please don’t say you’re still running TMG or ISA?
- WAC / OOS server. Don’t forget you need to update the farm with the new cert’s Friendly Name.
- Video-conferencing equipment.
- Have you embedded the cert in the config file for your VVXs or other Qualified devices?
- Are you doing any kind of Mobile Device Management to get your root cert to BYOD’s?
- Any other non-domain attached devices you might be talking TLS?
- How did/does the cert chain get to your Macs?
Have I missed any?
- I *always* reboot servers and SBCs after changing certs. Yes, 95% of the time it’ll just work, but for that remaining 5% things will go horribly, horribly wrong when the old one expires. In the middle of a business day.
- On your Front-End, Exchange and WAC servers, double-check the cert was automatically updated in IIS. The one time you don’t is the one time it won’t have.
- I’ve found that the Sonus 1k & 2k SBCs are inclined to misbehave if you let a root or intermediate cert expire in-situ. Delete the old one as soon as it’s no longer in use in your deployment, and before it expires.
- While you’re at it, if your Edge doesn’t have Internet access, now’s a good time to update its trusted root store. Thank you Pat, hat-tip David.
- If you’re replacing the OAuth cert or the public cert on the Edge, do this carefully, using the “-Roll -EffectiveDate” parameters. It’s well-documented here on TechNet.
- You’ve updated your internal Certificate Authority to stop issuing SHA1 certs haven’t you? Here’s how: “Transitioning Your PKI to SHA2″.
- There’s no point enduring this pain any more frequently than you absolutely have to. Chris Hayward walks you through “how to increase the internal certificate validity period“.
- Don’t forget: If you have a new Root certificate, make sure you’ve deployed it to all of the above BEFORE you start issuing device certificates against it.
- SfB Certificates Report: Guy Bachar & Yoav Barzilay (with some input from Anthony Caragol) – and now Amanda Debler – have crafted a fantastic script that will read your Lync/SfB topology and query all the servers (using WinRM) and gateways then prepare a report showing how long each of the cert’s still has to live.
- Test-CertificateStore.ps1: If your new OAuth cert isn’t replicating or your Front-End service won’t start, it might be due to bad cert placement. David Paulino‘s script to the rescue!
- Update-SfbCertificate.ps1: I wrote this having tired of the tedious and pain-staking requirement to check the SANs on the replacement cert all align with the original. I wrote it to be the Swiss Army Knife of certificate management for Lync and Skype for Business, and (amongst other things) it now provides you a way to effectively perform a “Get-CsCertificate | Request-CsCertificate” in one line – with the bonus that “Set-CsCertificate” (the ‘Assign’ step) is a right-click away at the end – I leave it on the clipboard for you.
- Compare-PkiCertificates.ps1: While I was writing the above, it occurred to me that the certificate comparison engine I was writing for it had wider application, so I spun it out into its own script. Feed this one two thumbprints and it will provide a quick on-screen comparison, with colours to show where they differ.
- Chad McGreanor’s walk-through on requesting a cert for TMG (and also applies to Web Apps).
- DigiCert Certificate Utility for Windows – “Certificate Management & Troubleshooting Made Easy”.
- Pat Richard’s New-ExpiringCertificatesReminder.ps1 will create a scheduled task and e-mail you as the certs in the Local Machine store are about to expire.
1st September 2016: This is the initial release.
2nd September 2016: Added Pat’s “New-ExpiringCertificatesReminder.ps1” to the Tools list.
29th September 2020: Updated the SfB/Lync Certificate Report link to its new home on GitHub.
“unfortunate incident back in June with the greiginsydney website”
Doesnt your public certificate provider warn you via email 2-3 months in advance regarding certs you bought from them that are about to expire???
Theoretically, yes. Mail to me reaches my on-prem Exchange 2013 via Exchange Online Protection (EOP). The expiry notifications never arrive. No idea where they’re going – it’s on the to-do list to take up with Digicert, but for the time being I now have a quarterly calendar reminder. ;-)
“I *always* reboot servers and SBCs after changing certs.”
When/why would you use -Roll with OAuth cert replacement if you ALWAYS reboot the server after replacing it anyway? Won’t the reboot end all connections to the server, and therefore the Roll feature is pointless?
OK you got me, John. I agree that was somewhat of a generalisation. You still want to -Roll your new OAuth and the public-facing Edge certs. In that case it would be the reboot that’s pointless, as the new cert has an “-EffectiveDate” in the future.