Few Gotchas! While Making Your Keycloak FIPS compliant

Keycloak is a very popular Java based open source Identity Access Management product. The recent versions (17.x onward) have gone server-less using Quarkus. When dealing with government, your SSO/ IAM system required to be FedRAMP and FIPS compliant.  Keycloak version 21.x+ is FIPS-2 certified, but additional configurations are needed to make the default Keycloak FIPS 140-2 enabled. 

Enabling FIPs mode in Keycloak is documented here. The documentation covers the steps in a greater detail. However, I have not seen any IAM deployed out of box, so the documented steps are not enough. I went thru series of hair pulling hours to make it work in my environment.  I am jotting down those gotchas here hoping this will help someone.

The FIPS standard revolves around the use of approved crypto module. Keycloak uses only FIPS approved crypto algorithm for it's functionalities when fips-mode is enabled.  

Keycloak recommends to use FIPs validated BouncyCastle library when strict fips-mode is enabled.  So the starting step is downloading BouncyCastle-FIPS bits and adding them to the Keycloak distribution as a crypto Service Provider, so BCFIPS bits will be used instead of the default bits.

(1) Download BouncyCastle jars

The Keycloak documentation instructs to download the following BouncyCastle jars and add them to KEYCPLAO_HOME/providers directory.

bc-fips-1.0.2.3.jar, bctls-fips-1.0.14.jar, bcpkix-fips-1.0.7.jar

However, by copying these jars to "providers" directory does not register it as providers unless you build the Keycloak again. Here is an example of the code snippets from my Dockerfile.

:

ENV KC_FEATURES=fips

ENV KC_FIPS_MODE=strict

ENV KC_HTTPS_PORT=8443

ENV KC_HTTPS_KEY_STORE_TYPE=BCFKS

ENV KC_HOSTNAME="sso.poc.abc.com"

:

ARG BOUNCYCASTLE_FIPS_JAR_LOC='https://downloads.bouncycastle.org/fips-java'

RUN curl -qsSL --retry 3 --retry-delay 0 ${BOUNCYCASTLE_FIPS_JAR_LOC}/bc-fips-1.0.2.3.jar -O --output-dir /opt/keycloak/providers &&\

    curl -qsSL --retry 3 --retry-delay 0 ${BOUNCYCASTLE_FIPS_JAR_LOC}/bctls-fips-1.0.14.jar -O --output-dir /opt/keycloak/providers &&\

    curl -qsSL --retry 3 --retry-delay 0 ${BOUNCYCASTLE_FIPS_JAR_LOC}/bcpkix-fips-1.0.7.jar -O --output-dir  /opt/keycloak/providers &&\

    chmod +x /opt/keycloak/providers/*.jar

RUN /opt/keycloak/bin/kc.sh -v build

The build (last line is) needed to register the providers

(2) Use of BouncyCastle Keystore :

By default, Keycloak uses "PKCS12" keystore and the same is also supported for BCFIPS non-approved mode. However "BCFKS" (BouncyCastle key store) is a must when keycloak runs in strict fips mode. BCFKS can be generated various way (from scratch or from existing PKCS12 etc) depending on your use case.

> Use case I: Creating BCfKS from scratch

Set the secure random algorithm. This will be used in the next step.

$ echo "securerandom.strongAlgorithms=PKCS11:SunPKCS11-NSS-FIPS" > kc.keystore-create.java.security

$ keytool -genkeypair -noprompt\

  -sigalg SHA512withRSA \

  -keyalg RSA \

  -alias sso.poc.server \

  -storetype bcfks \

  -dname "CN=sso.poc.volterra.us, OU=f5xc, O=f5, L=San Jose, S=CA, C=US" \

  -keystore server.keystore.bcfks \

  -storepass ChangeMe! \

  -keypass ChangeMe! \

  -providername BCFIPS \

  -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \

  -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \

  -providerpath ./lib/bc-fips-*.jar \

  -J-Djava.security.properties=kc.keystore-create.java.security

$ cp server.keystore.bcfks /opt/keycloak/conf/server.keystore

> Use case || : Converting an existing PKCS12 file to BCfKS
This is a typical use case when you already have a PKCS12 keystore delivered by your PKI team ot some other means. The PKCS12 may have been generated using opnenssl or java keytool. 

Example of generating PKCS12 using openssl

$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 3650 -node -subj "/C=US/ST=CA/L=SanJose/O=security/OU=sso/CN=sso.poc.abc.com"

$ openssl pkcs12 -export -nodes -password pass:"" \

 -inkey key.pem -in cert.pem \

 -name "server" \

 -out server.keystore.pkcs12

Example of generating PKCS12 using Java keytool

$ keytool -genkey -noprompt \

 -keyalg RSA \

 -alias server \

 -storetype PKCS12 -validity 3650 -keysize 2048 \

 -dname "CN=sso.poc.abc.com, OU=sso, O=security, L=San Jose, S=CA, C=US" \

 -keystore server.keystore.pkcs12 \

 -storepass ChangeMe! \

 -keypass ChangeMe!

The aboves are just example of PKCS12 keycstore creation. The assumption here is, you already have a PKCS12 keystore. Now convert the PKCS12 created above to a BCFKS. 

$ keytool -importkeystore -v \

    -srckeystore server.keystore.pkcs12 \

    -srcstorepass "" \

    -srcstoretype PKCS12 \

    -srcalias "server" \

    -srckeypass "" \

    -destkeystore server.keystore \

    -deststoretype bcfks \

    -deststorepass "password" \

    -destalias "server" \

    -destkeypass "password" \

    -providername BCFIPS \

    -providerpath ./lib/bc-fips-*.jar \

    -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider \

    -J-Djava.security.properties=kc.keystore-create.java.security

# test the BCFKS Keystore that is just created by listing it"

$ keytool -list -v -keystore server.keystore \

  -storepass "password" -storetype bcfks \

  -providerpath ./lib/bc-fips-*.jar -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider

$ cp server.keystore.bcfks /opt/keycloak/conf/server.keystore

My use case was similar to above, but little twisted.  My PKCS12 file was delivered by another centralized PKI/ Keystore service through a side car, which runs in my container. So my BCFKS keystore was build from this sidecar provided PKCS12 during my Keycloak container building process.  In this scenario, the above code did not work. I get this error while building BCFKS using keytool.

Importing keystore server.keystore.pkcs12 to server.keystore...
Exception in thread "main" java.lang.IllegalAccessError: class org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider$CoreSecureRandom (in unnamed module @0x4f2d995e) cannot access class sun.security.provider.SecureRandom (in module java.base) because module java.base does not export sun.security.provider to unnamed module @0x4f2d995e
	at org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider$CoreSecureRandom.<init>(Unknown Source)
	at org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider$4.run(Unknown Source)
	at org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider$4.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)

The reason is, the Bouncycastle (bc-fips-1.0.2.3.jar) is not certified for JDK 18 (As of Sep 2023,).  So I had to rebuild my image with JDK 11.

:

RUN microdnf install -y java-11-openjdk-headless ....

:    

RUN /opt/keycloak/bin/kc.sh -v build

:

This fix works in Keycloak version 21.X, but may break in version 22.X. I did not cover all detailed steps as the deployment architectures varies. I tried to jot down the critical blockers that slowed me down in this project.  

Thanks for stopping by my blog!

Painfull process of Renewing IDP's Signing Certificate in SAML Integration

I learned my first SAML integration in 2005 using a product called SHARE-id developed by Oblix that was later part of IAM product suite of Oracle. Last 10 years there is a huge adaption of SAML protocol for a seamless integration with a centralized Authentication service providers like Azure, Ping, Forgerock, Google and so on. You can also write your own Authentication service supporting SAML. Its is easy to find free library to implement a SAML IDP (Identity Provider) or SAML Assertion consumption service for a SP (Service Provider). 

Here is a diagram of a standard SAML flow.

 

I will not describe how SAML integration work as many good articles are already available in the net. This write-up specifically focus on "Certificate Renewal" aspect of SAML integration. It is one of the major pain point in SAML integration. 

Context  :- 

One or many Service-Providers (SP/ web application) tied to a Identity-Provider (IDP/ Azure, Forgerock, ping, home-grown) in a “Circle of Trust”, where IDP shares the certificate with all SPs.  SP uses this certificate to verify the digital signature in SAML-Response that proves that it has indeed come from the chosen IDP.

 

Now the Issue :- 

Just like any other certificate, IDP’s certificate also expires every 3yrs. All the participating SP need to replace the expired cert with the new one.  When you have 100+ SAML apps in production,  co-ordinating with so many AppOwners and getting the cert replaced requires good amount of planning and effort

Possible Solutions :- 

(1) Plan it out in advance :- 

This is the bare minimum an IDP owner can do :-

o  Add a new certificate to each IDP in non-prod env 3 months before the expiry.  Coexistence of “to-be-expired” certificate with “new” certificate ensure no outage.

o The new cert (and/or IDP-metadata) then distributed to each AppOwner. 

o The expired certificate is removed from the IDP once all AppOwners confirm that their app's SAML integration is working with new certificate.  

o Same processes  followed in production. 

(2) Can we improve this process and automate it ?

Yes, all depends on your organization budget though :)  An IDP-Application-dashboard portal can be built where App-Owners can seamlessly onboard their SAML compatible application thru user friendly App registration process. 

o IDP owner proactively add the new certificate to the IDP before the expiry and portal then notify the app-owners.

o App-owners picks up the certificate and DP meta data and test the SAML integration.

o The portal then remove the old certificate from IDP immediately after app-owner certify the successful integration using the new certificate. 

Basically, the portal replace the the manual co-ordination between IDP-Owner and App-Owner by automating the IDP cert renewal process.  

(3) Any other Options?

Yes, switch to OIDC based integration, which is based on light weight token compared to XML heavy SAML response. OIDC token is signed using either a client-secret or jwks (Json Web Key). We will cover this in the next blog.

 

 

Practical Tips To Stay Focused In An Ever Distracting Environment

The companies are increasingly adapting Agile methodology, Continuous Integration and Development process to make the delivery efficient.  There is also a noticeable increase of productivity among the developers.  The management expectation has gone up and this creates a quite a bit of stress  on engineers to keep the pace. Staying focused on finishing one logical task becoming increasingly challenging for Lead Engineers when they handle multiple modules with a baggage of supporting many of their previous projects.

Here are few practical tips on staying focused on your high priority tasks.

1. Start your day addressing the Critical item only , not reading the email :-
Many of us starts our day going through the new emails and replying them.  I will suggest spending first two hours of your day addressing the P1 items like :-

• Any P1 follow-up items from yesterday. Note that you will use the email only for this purpose, not reading the new emails.
• Any P1 technical issues/ code you are working on.

This will give you a grip over all P1 items.  Reading new emails will distract you to new set of works, rather address the issues that bothering you.


2. Allocate specific time for new email :- 
So when should you read your new emails? Allocate two specific times in a day for new emails and replying them. For example :-

• Around 10am, two hours after you finish your critical items. 
• Again at 4pm, before winding down. Now you may have a TO-DO list for tomorrow from these emails.

3. Dealing with too many emails :- 
A very simple solution for this is to apply effective email-filter, so only the relevant email stays on the top.  All enterprise email program comes with filter utilities.
Here are some filter examples :-

• All emails where you are CCed should goes to "NotToMe" folder . The reason is obvious, you don't have to reply it immediately.
• We receives tons of email system generated email like, "server xxxx restarted", "The Certificate xxx is about to expire", etc.  Carefully choose these regular emails that belongs to trash. 
• Put a similar filter to all regular blah..blaha email straight to trash.
• Create sub folder for different subcategories.  For example anything coming from Jira should go to Jira subfolder.

Building a custom list of  email-filter will take time as it need constant adjusting. But this is one of the most critical tools that helps you avoid wasting time on non critical items.


4. Block few hours for Technical works :- 
The Senior Engineers spend more time on co-ordination and techno clerical work, thus loosing the focus on core technical work.  The best way to address this issue is block at least 16 to 20hrs of your calendar for development work.


5. Everything has a home :- 
Many external factor helps you stay focused.  Few examples could be :- A clean desk, decluttered workspace, clean email box etc.  How to have a clean desk ?  I got this key take away statement "Everything has a hone"  from a time-management training. Any object lies on your desk should have a home. The home could be a paper tray, drawer, book shelves, trash can etc.  Same goes for email.  Every email has a home (a rule based folder) .

6. Continuous Planning :- 
This is mostly applicable to Senior / Lead engineers who are very often face the questions like when it will be done? Why it is not done? What taking so much of time ? What is the estimation ? and so on .. So continuous planning is THE critical task, that will provide you the tool that will help you answering these questions.

Some useful tips on planning tools :-

• TO-DO list for the Month/ Quarter - Use tools like a formula based planning spreadsheet or Jira structure that will be handy in getting a near perfect report on the project status. Particularly, the number of remaining-hours required for your project stories. At-least 2 hrs a week need to be spend in updating the story detail. Usually I reserve 2 hrs every alternative Friday. 
• Ensure that your Monthly plan match the road map time line done by your manager. 
• TO-DO list for the Week :- Maintain a sticky note or a working journal for listing every-days TO-DO list.  This takes only 10 mins and can be done in the morning.  Plan your day around this weekly TO-DO list.

Conclusion :- 
I learned these over the years working in a very demanding environment.  I try my best to stick to these rules, though I just do fire fighting in some of the chaotic days. These tips are not new to many of you.  Love to hear your experience on dealing with distractions and how to stay focused. 

Mistaking Kerberos as Single Sign On

End users often misinterpret the Kerberos authentication mechanism to mean Single Sign-On (SSO) in corporate world.  Kerberos authentication provides seamless sign on to (web) applications by using the Kerberos ticket generated/ refreshed during the user’s authentication to the network (when user first login to the computer).

For example, if web applications AppX and AppY are configured for Kerberos authentication, the user will not see any credential challenge when both of these apps are accessed in two different tabs of same browser. This gives the impression of SSO between AppX and AppY.

The SSO functionality is much broader in any in premises Enterprise SSO infrastructure like OAM, SiteMinder, ForgeRock. The Kerberos mechanism is just one of several Authentication schemes available in SSO server.  The Other authentication scheme like Form Authenticaiton can also provide single sign on between two different web apps thru a secure cookie generated by sso server when user first login thru the Form in a Web-Agent based SSO architecture.  

OAM R2 Persistent Login (Keep Me logged In/ Remember Me) Set up

I remember struggling with this one in OAM R1.  Since this feature was not available in R1, we implemented this by our own custom code.  Finally this is available in R2.  Here is the link (http://docs.oracle.com/cd/E40329_01/admin.1112/e27239/getstarted.htm#CIHBEEAI) to set this up and test in 30 minutes max.

I am just posting the 1st step of exact WLST command taht I ran to enable it in OAM 11g R2.  Follow the rest of the steps from Oracle doc


$ cd ../Middleware/Oracle_IDM1/common/bin
$ ./wlst.sh

wls:/YOUR_DOMAIN/serverConfig> connect()
Please enter your username :weblogic
Please enter your password :
Please enter your server URL [t3://localhost:7001] :t3://YourServer:7001
Connecting to t3://YourServer:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to YOUR_DOMAINr'.

wls:/YOUR_DOMAIN/serverConfig> configurePersistentLogin(enable="true", validityInDays="30",maxAuthnLevel="2", userAttribute="obPSFTID")
SUCCESS

wls:/YOUR_DOMAIN/serverConfig> exit()

Including Custom MANIFEST.MF File in NetBeans jar Building Process

OAM custom authentication module requires a custom manifest.mf file to be included in your custom plugin jar file. It is is easy to include your own manifest.mf file during jar building process in eclipse. But it not that easy if you are using Netbeans .  If you do a clean build using Netbeans, you will end in having a default manifest.mf file in the jar file.

Here are the steps to include your OAM plugin specific manifest.mf file in the jar.
(1)  Keep your custom manifest.mf file in /src/META-INF/ directory.
(2) Open /YourProject/nbproject/project.properties
comment this line
#manifest.file=manifest.mf

add this two lines that will do the magic
manifest.available=true 
manifest.file=${src.dir}/META-INF/manifest.mf 

(3) Now do a NetBean build, the generated jar file should have "/src/META-INF/manifest.mf" included already.

Japanese (non-ASCII) Character Issues in OAM headers variable

I have few user records with Japanese character (like 高山) in their attributes. Querying these users in OVD through ODSM displays it correctly . However, the same attributes when passed as OAM header variable through OAM-Authorization-Scheme spit out garbage like '=?UTF-8?B?6auY5bGx?=' .

This happens because the Webgate will encode any non-ASCII characters in header variables according to RFC 2047. It is the receiving OAM-protected application that will decode the encoded characters. The following java sample code will be a quick rescue. This will transform the non-ASCII character to a readable one.
:
import javax.mail.internet.MimeUtility ;
System.out.println("The Actual Value is := " + MimeUtility.decodeText("=?UTF-8?B?6auY5bGx?=")) ;
:

Your result will be =
The Actual Value is := 高山