This weekend I was involved in rolling over the ADFS Token Signing and Token Encryption certificates while a huge amount of application were connected using WS-Federation, SAML or OAuth. With regards to rolling over the certificates, only the WS-Federation and SAML based apps are impacted, not the OAuth apps.
Short answer: preparation, communication and automation are KEY!
–
This ADFS system was designed and implemented by me years ago. In the beginning, there were just a few apps and it grew gradually. After a moment it was starting to grow like crazy. More and more app devs smelled the federated SSO cool stuff and wanted that for their app(s). Why? Default globally supported protocols between organizations, apps, etc, and all based on federation. Supported by ADFS, many other on-premises federation systems and cloud based systems, such as e.g. Azure AD. The focus of this blog post is apps connected to ADFS as the federation provider, for both downstream identity providers and upstream service providers.
–
When I design this system, after the technology, I also designed the required processes to make sure the technology was used in the correct way. If you do not design the processes in addition to the technology, it becomes a mess and because certificates are used it will drop dead, exactly 1 year after installing/configuring it. Why will it drop dead after 1 year? The default Token Signing and Token Encryption certificates are ADFS managed and expire after 1 year. All apps not (being) able to consume to the federation metadata URL automatically will drop dead if no action is taken in time! And rushing with no process in place is a good cocktail for possible failure. If you have a handful of apps, you should be OK, but if you have 100+, or 200+ or even more apps connected to ADFS, you seriously need a process to get this done in time and correctly. Regarding the process, that ADFS has internally, please see: (2013-05-14) ADFS Managed Certificates Supporting Auto Certificate Rollover
–
Processes:
- Backup/Export of the complete ADFS configuration
- Provisioning of applications onto ADFS
- Recertification of any connected application, keep or deprovision
- Notification of expiring certificates in ADFS (about 3 months ahead of time)
- Certificate change in ADFS
- Validation of specified e-mail addresses against AD
–
During the onboarding of ANY application the following was recorded:
- Does an application support automatic consumption of the federation metadata URL or does it support its consumption manually through an XML file?
- Does an application support just 1 or 2 Token Signing certificates from an identity provider?
- When an application supports the consumption of the federation metadata file, AND it supports only 1 Token Signing certificate, which certificate will it consume when it reads the federation metadata file as it contains multiple Token Signing certificates from the identity provider
- E-mail addresses from at least 3 “owners”
- E-mail address of either a functional mailbox or distribution list with a support team behind
–
[AD.1]
If it supports automatic consumption of federation metadata through a URL (for ADFS: /federationmetadata/2007-06/federationmetadata.xml">https://<federation service FQDN>/federationmetadata/2007-06/federationmetadata.xml) it processes the information automatically. If it does not, you may need to first export the federation metadata to an XML file to be able to import it into the application. Even though the federation metadata URL is used, it does not mean an application uses it automatically on a regular basis (e.g. every day). It is possible that a admin need to run a script, or push some button or whatever
–
[AD.2]
If an application supports 2 Token Signing certificates from an identity provider, a smooth certificate rollover is supported without ANY pain. If only 1 Token Signing certificate is supported it means, the certificate must be replaced at (nearly) the same time as the ADFS admin switches the primary and secondary Token Signing certificate
–
[AD.3]
This scenario should be very rare, but unfortunately some apps do it incorrectly. When the federation metadata from the identity provider is imported, and only 1 Token Signing certificate is used, and the federation metadata from the identity provider contains multiple Token Signing certificates, which is consumed (used) by the application? The first Token Signing certificate occurrence, or the last occurrence? If it is the last occurrence, then that is good, but the federation metadata XML file must only be imported at the same time when the switch is made in ADFS. If it is the first occurrence, then the app is not following best practice and it will break as soon as the switch in ADFS is done.
ADFS by default:
- Publishes ALL Token Signing certificates in the federation metadata
- The oldest Token Signing certificate is published first and the newest Token Signing certificate is published last
- Switching Token Signing certificates in ADFS DOES NOT change the order of the Token Signing certificates in the federation metadata
- Publishes only the primary (active) Token Encryption certificate in the federation metadata
- Switching Token Encryption certificates in ADFS removes the old Token Encryption certificate from the federation metadata, and adds the new Token Encryption certificate to the federation metadata
- ADFS signs the federation metadata with the primary (active) Token Signing certificate
So if the first occurrence of the Token Signing certificate is consumed, after the switch in ADFS, the only way to solve this is:
- Edit the federation metadata XML file and remove the first occurrence of the Token Signing certificate (old Token Signing certificate, marked as “use=’signing’”, appears 3 times in the federation metadata XML), and keep the last occurrence (the newest) Token Signing certificate
OR - Remove the oldest Token Signing certificate as a secondary from ADFS, export the federation metadata to an XML file and import it into the application
Which option you use depends on whether or not an application checks for the signing of the federation metadata. If it does not, you can use either option, but it is easier to edit and you can keep the old Token Signing certificate as secondary if for whatever unlikely reason want to rollback (not recommended when having many applications (define “many”!) as it may turn out in a logistics nightmare
–
[AD.4]
A minimum of three e-mail addresses are required to support the yearly recertification process where a decision is made to either keep or deprovision the federation trust in ADFS. A minimum of three mail addresses as people come and people go (internally or externally), and chances are very low that all three people move at the same time (unless there is a reorganization, sigh!). If any of the listed people changes roles
–
[AD.5]
The e-mail address of a functional mailbox or distribution list with people from a (technical) support team behind it. This is mainly used by the “Certificate Change in ADFS” process. The chance of failure must be brought to a minimum by not having dependency of e-mail addresses from people, but rather have e-mail address of a functional mailbox or distribution list with people behind it. The people from a (technical) support team are responsible to updating the application with the new Token Signing when ADFS changes the certificate.
–
Now having all this information, you need to be able to store it somewhere that makes it easily manageable and provides the ability to process it through scripting (automation). Within ADFS, every single trust has a notes or description field, and the field can perfectly be used to store this information. However it is a string formatted field with a maximum of 3000 characters. I thought of the idea to implement an XML structure to be able to read from and write to it as needed. Works like magic as we can script against it too. But how about using CMDB? Well, we wanted to be certain of the up-to-date-ness of the information, we wanted to be able to script against it and we wanted it as close as possible to the trust configuration in ADFS.
–
Now, the whole certificate change in ADFS looks like something similar to:
- Define "Product Backlog Items (PBI)" on the backlog, plan into sprints and assign to people
- Organize initial meeting with colleagues to explain process, answer any questions and discuss PBIs
- Plan meeting with (higher) management and deliver a presentation about the certificate change, discussing what, when, why, how, current state (number of trusts), risks, impact, rollback (none!), etc.
- Raise an RFC to change certificates in ADFS and push to achieve the highest risk/impact category possible
- Check and fix if needed e-mail addresses from application owners and support teams
- Request new certificates for ADFS (SSL, Token Signing and Token Encryption)
- Install all new certificates in the personal certificate store of all ADFS servers
- Install the new SSL certificate in the personal certificate store of all WAP servers
- Configure the new Token Signing certificate as a secondary Token Signing certificate
- Configure the new Token Encryption certificate as a secondary Token Encryption certificate
- Configure the new Svc Comm/SSL Certificate on all ADFS servers and WAP servers
- Recheck and fix if needed e-mail addresses from application owners and support teams
- Send the "First Notification" to all support teams and application owners
- Notify the global service committee about the upcoming change, provide information and request to chase teams/owners
- Send the "Reminder Notification" to all support teams and application owners
- Switch the Token Signing/Encryption Certificate in ADFS
- Send the "Switch Executed Notification" to all support teams and application owners
- Guide and support support teams and application owners that request or needed guidance
- Organize a closure meeting with colleagues to discuss any feedback from support teams and application owners, colleagues involved, and if applicable possible improvements for the process
- Process the feedback and improvements from the closure meeting for the next time
–
End Result: SUCCESS! Oops I did it again!
–
I have done this multiple times now, and even though there is nothing to it (that’s what I think!) it is always exciting due to the huge amount of applications involved! The most difficult part is explaining this to non-technical people where you have to say things like: “there is no rollback”, “there will be issues with apps”, “fix forward only”, etc. It just scare the crap out of them and I understand that. Another one not to forget is, the support team, and sometimes even the vendor or service provideers, managing the application do not have the technical knowledge regarding federation and SSO
–
But, been there, done that, got the t-shirt, all sizes and all colours!
–
–
Cheers,
Jorge
————————————————————————————————————————————————————-
This posting is provided "AS IS" with no warranties and confers no rights!
Always evaluate/test everything yourself first before using/implementing this in production!
This is today’s opinion/technology, it might be different tomorrow and will definitely be different in 10 years!
DISCLAIMER: https://jorgequestforknowledge.wordpress.com/disclaimer/
————————————————————————————————————————————————————-
########################### Jorge’s Quest For Knowledge ##########################
#################### http://JorgeQuestForKnowledge.wordpress.com/ ###################
————————————————————————————————————————————————————-