Summary

This post summarises the effort going from Google KMS to Azure. We are migrating from Google Apigee and Google KMS to Azure AD, Azure B2C, Key Vault, and Notification Hub for authentication and push notifications in our new mobile app. Users will register through Azure B2C, and their device ID will be stored and registered with Braze. Push notifications will be managed via AWS Firebase for Android and Apple APNs for iOS. The On-Behalf-Of (OBO) Flow is still in preview, so it will not be used at this time.

A key part of this setup is certificate-based authentication to securely pass bearer tokens between services. In this post, I’ll demonstrate how we generate a private key and a Certificate Signing Request (CSR) using OpenSSL, then sign it with a self-signed root CA. The final signed certificate is converted into a PFX (.pfx) file, which is required for storage in Azure Key Vault. This ensures the private key remains protected while allowing secure authentication between platforms.

For secure API authentication, we generate JSON Web Tokens (JWTs) signed with the private key stored in Azure Key Vault. The JWT header includes a Key ID (kid) or X.509 thumbprint (x5t), and the payload contains claims such as issuer (iss), subject (sub), audience (aud), and expiry (exp). The unsigned JWT is sent to Azure Key Vault, which signs it before it is used for authentication. The receiving API verifies the signature, expiry, and audience claim before granting access, ensuring a secure and scalable authentication process.

The Problem Context

Our platform is responsible for delivering over a million push notifications per day, but without a secure authentication mechanism in place, we always seem to face challenges in ensuring reliable delivery. The chart below represents real-world data on push notification scaling, highlighting the impact of a missing cohesive authentication flow.

Without a proper authentication mechanism, tokens frequently fail to validate as they move between services, leading to repeated delivery failures. This results in high latency, increased retry attempts, and unnecessary resource consumption. The red line in the chart represents the notifications that failed to be delivered due to authentication issues, while the grey dashed line indicates the total notifications that should have been sent.

The lack of a systematic authentication system causes push notifications to be rejected at various integration points, reducing the efficiency of our messaging pipeline. This not only affects service reliability but also increases the operational burden on infrastructure and support teams.

Poor Performance Due to Authentication Failures

The failure rate due to poor authentication is significantly higher, reaching 30% of all push notifications. Over the course of a day, where 1,000,000 notifications are expected to be delivered, this translates to 300,000 failures, leaving only 700,000 successfully processed messages.

To break this down further:

• If notifications are being sent evenly across 24 hours, the system should be delivering approximately 41,667 notifications per hour.
• However, with a 30% failure rate, only 29,167 notifications per hour are actually reaching users, while 12,500 notifications per hour fail.
• At peak hours, when the platform attempts to push 100,000 notifications per hour, failures skyrocket to 30,000 notifications per hour, significantly impacting reliability.

This seems to be a common design issue or a fault of Google’s KMS, it only seems to support a handful of authentications flows. This also means that you need to monkey-patch workarounds to get what you need. I have another post where I actually demonstrate how to monkey patch authentication requests in Google KMS; however for this post, we’re going to officially build out a vault system with Microsoft’s help.

Generate a Private Key and a Certificate Signing Request

In this example, I’ll be using OpenSSL to create a Private Key, The CSR and sign it assuming the position of being a Certified Authority (this should not be done in production).

Generate a Private Key

The command below generates a Private key called private.key. You will be prompted to add a passphrase with the optional flag -aes256, which encrypts the private.key. Note that at the time of writing (February 2023), both Google Cloud Key Management (KMS), and Azure Key Vault (AKV) do not make use of the this passphrase, this has to be implemented in your programming logic.

openssl genpkey -algorithm RSA -out private.key -aes256

A common question I get asked a lot; why don’t we just encrypt using the RSA algorithm? The reason is quite simple – it’s just not practical and not designed for large content.

The command above generates the private key using the RSA algorithm, this is then encrypted using AES. The output should look as follows:

openssl genpkey -algorithm RSA -out private.key -aes256
...+++++++++++++++++++++++++++++++++++++++*...+++++++++++++++++++++++++++++++++++++++*..........+.....+.......+.....+..................+...+....+.....+.+.................+..........+...........+....+......+.........+.....+...+.......+..+....+..............+.+..+...+................+......+..+...+...+....+.....+.....................+......+.+..+.......+...+.....+.+.........+...+.....+.......+..+....+.....+.+.....+...............+..........+...+..+....+...........+.+..+.+..............+.+......+...............+...+..+.+..+...............+...+..................+.......+.....+...+....+...+..............+......+.......+...........+.+...+...+.................+..........+..+.+..+...+...+.+...+..+....+..+...+.........+.......+...+...............+...+.....+...+.+...+..+...+.............+........+.+......+.....+.........+.+...............+.................+...+.............+.....+......+....+..........................+.+...............+.....+...+....+......+.................+.......+.....+......+.........+......+.+.........+.....+.+......+.....+....+.....+......+...+......+...+.......+...+..+.............+.....+...+..........+...+.....+...................+............+......+..............+..........+...+..+...+.+...............+..+....+......+..............+...+...+.........+.+........+.......+.....+...+....+...............+..+....+.........+...+...+......+.........+.....+......+...+.+............+..+......+......+......+...............+............+.......+......+..+.+.........+..+.+.................+.+.........+...+........+.+...+...+............+..+.............+.....++++++
.+..........+.....+.+..+............+......+++++++++++++++++++++++++++++++++++++++*..+.......+...+++++++++++++++++++++++++++++++++++++++*..........+...+...+.+...+...+.....+....+......+..................+.....+.+...+..+.+.....+.......+...............+..+.+..+.+......+.........+..+...............+...+.+..............++++++
Enter PEM pass phrase:

Verifying - Enter PEM pass phrase:

Generate the CSR

The next step is to generate a Certificate Signing Request (CSR). This is established using the following command:

openssl req -new -key private.key -out request.csr

openssl req -new -key private.key -out request.csr
Enter pass phrase for private.key:

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:UK
State or Province Name (full name) [Some-State]:England
Locality Name (eg, city) []:London
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Eax360
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:eax360.com
Email Address []:shussain@eax360.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:robot
An optional company name []:

Note: The Common Name (CN) should be the domain name or entity name you’re securing

Create Your Own CA (Self-Signed CA)

To act as your own certificate authority (CA), you need a root CA key and certificate.

Generate a Root CA Private Key

openssl genpkey -algorithm RSA -out rootCA.key -aes256

Again, choose a passphrase.

Create a Root CA Certificate

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.crt

This will prompt you for CA details.

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.crt
Enter pass phrase for rootCA.key:

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:UK
State or Province Name (full name) [Some-State]:England
Locality Name (eg, city) []:London
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Eax360
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:eax360.com
Email Address []:shussain@eax360.com

At this point, you should have four files in this directory:

We will now proceed to simulate a CSR with the CA.

Sign the CSR with Your CA

Now, sign your CSR (request.csr) with your root CA to issue a certificate.

openssl x509 -req -in request.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out signed_certificate.crt -days 365 -sha256

This signs the CSR using your own CA.

The resulting signed_certificate.crt is now a self-signed certificate.

When signing a Certificate Signing Request (CSR) with OpenSSL using a Root CA, a file called rootCA.srl is generated. This file stores the serial number used for the issued certificate, ensuring each certificate gets a unique identifier. OpenSSL automatically increments this number for subsequent certificates signed by the same CA. If rootCA.srl is deleted, OpenSSL will not know what serial number to use next, potentially causing conflicts. To view the details of a signed certificate, you can use openssl x509 -in signed_certificate.crt -text -noout.

Verify the Certificate

You can check the generated certificate details with:

openssl x509 -in signed_certificate.crt -text -noout

The output should look something like this:

E:\Experiments>openssl x509 -in signed_certificate.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            63:52:f4:0d:23:05:72:6f:5d:e3:05:15:3e:6e:c1:fb:b6:b3:71:11
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=UK, ST=England, L=London, O=Eax360, CN=eax360.com, emailAddress=shussain@eax360.com
        Validity
            Not Before: Mar 20 09:10:48 2025 GMT
            Not After : Mar 20 09:10:48 2026 GMT
        Subject: C=UK, ST=England, L=London, O=Eax360, CN=eax360.com, emailAddress=shussain@eax360.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bb:6f:86:03:80:72:c0:36:5e:1d:83:55:4f:0a:
                    96:6b:c3:80:86:fa:f8:77:7a:16:f3:a8:38:be:6a:
                    a2:3f:d9:53:e9:6d:21:81:ce:14:af:e7:14:84:9b:
                    16:0d:8e:32:06:1b:38:65:62:05:4f:ac:29:36:b8:
                    e8:ca:e1:7f:89:c1:17:4d:2e:1b:aa:dc:45:a4:5b:
                    47:dd:f7:c7:10:20:4e:50:3c:a8:79:eb:2e:92:9b:
                    65:9c:24:82:6f:21:28:28:58:14:08:34:28:29:fb:
                    2f:3a:f2:8b:9f:31:7d:59:02:ac:51:e1:00:43:54:
                    20:9d:9f:5e:4c:eb:49:3f:c4:b6:b3:b8:3a:90:b6:
                    39:41:dd:6e:45:08:ae:8f:89:f6:17:3a:aa:3d:e5:
                    b5:54:66:bd:a3:0f:ae:d7:2d:75:5f:df:f4:d1:d4:
                    99:fb:43:dc:1d:ff:83:e6:a3:8b:18:46:3f:49:45:
                    2d:d7:38:a9:c2:2e:c5:78:1c:93:7a:dd:ef:a6:37:
                    95:10:1f:22:66:d2:35:40:85:8d:13:4a:51:ce:bf:
                    b9:49:f0:77:6f:12:9c:68:08:f3:22:88:6c:44:8c:
                    33:5c:77:69:f0:63:c6:0e:f2:2d:6f:cc:78:a8:bc:
                    6b:1c:13:7a:ca:35:b2:7b:a6:35:3a:21:a6:bc:3f:
                    72:c7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                B6:B4:E4:67:8D:AD:92:C5:E8:5F:AA:57:89:F6:0A:96:1D:7D:1E:41
            X509v3 Authority Key Identifier:
                C3:D1:C3:56:C7:66:E9:C0:3C:AB:E4:42:71:88:3C:BF:BC:3E:D1:F0
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        30:93:2c:49:b0:ae:13:14:39:f4:5f:fa:ee:72:16:4c:19:6e:
        67:1d:9c:9e:b8:b4:7a:1c:36:f3:08:46:28:b1:3a:5a:c6:07:
        49:94:9d:c3:ab:43:16:0c:79:69:f3:d3:23:96:7a:f5:8a:0e:
        ff:85:cf:fc:c5:98:8b:26:28:60:19:5f:81:aa:ac:00:c0:54:
        d3:6b:74:4f:85:05:45:7a:50:fc:6b:67:56:82:25:aa:48:bf:
        79:e9:a7:38:b6:48:01:72:30:b4:1f:9c:c6:68:43:40:26:9f:
        2c:84:13:86:46:78:fe:07:6e:c7:98:39:0a:a5:62:b1:18:24:
        fb:a0:72:c8:e3:fc:b2:42:8d:0c:d5:3b:18:fe:ff:bf:13:46:
        de:c2:76:46:e1:78:47:e9:0e:a1:aa:21:80:03:25:8e:51:8c:
        7f:f4:48:e6:fb:f5:0d:31:a4:a4:76:0d:e4:99:55:82:b4:91:
        5a:ef:3e:92:79:3a:be:e3:4d:a6:ff:9d:5a:a0:17:33:df:78:
        97:eb:3d:84:8f:19:6e:39:53:58:80:20:6f:02:b3:1c:79:b9:
        8f:63:4b:9f:55:34:e6:bb:69:4f:67:6d:41:4e:2a:7f:7b:6b:
        78:9d:ba:02:a5:12:95:a4:62:ce:0e:64:88:36:6f:ef:c0:a3:
        9d:d1:30:e4

Summary of Output Files

• private.key → Private key for CSR
• request.csr → Certificate Signing Request
• rootCA.key → Your CA’s private key
• rootCA.crt → Self-signed CA certificate
• rootCA.srl → Stores the serial number of the generated certificate
• signed_certificate.crt → The final signed certificate

Convert to Azure Supported Format

Since Azure requires a PFX (.pfx) file, you need to generate one that contains both the signed certificate (.crt) and the private key (.key). Here’s how to do it:

.PFX (PKCS#12) – Contains the certificate and its private key. Used when you need to import both into Azure Key Vault. .PEM – Typically contains the public certificate and can also include the private key.

Export without Private Key

openssl pkcs12 -export -out signed_certificate_noPkey.pfx -inkey private.key -in signed_certificate.crt -certfile rootCA.crt

Export with the Private Key

openssl pkcs12 -export -out signed_certificate.pfx -inkey private.key -in signed_certificate.crt -certfile rootCA.crt

• -export → Creates a PFX file.
• -out signed_certificate.pfx → The output PFX file.
• -inkey private.key → The private key.
• -in signed_certificate.crt → The signed certificate.
• -certfile rootCA.crt → Includes the root CA certificate (optional, but recommended).

You’ll be prompted to set an export password—remember this password because you’ll need it when uploading the certificate to Azure.

Creating a Secure JWT Assertion with Azure Key Vault

This part of the post, explains how to create a secure JSON Web Token (JWT) using a certificate stored in Azure Key Vault. The process ensures that your JWT is properly signed and can be verified by the recipient without exposing your private key.

Retrieve the Certificate and Understand Its Key Components

First, you need to request the certificate from Azure Key Vault. This certificate provides the public details needed for your JWT, such as the Key ID (kid) and the x5t (X.509 thumbprint – this is used by Azure AD, kid is often used by other IDPs). The Key ID is a unique identifier for the key stored in Key Vault, while the x5t is essentially a fingerprint of the public certificate. Both pieces of information help the receiver determine which public key to use for verifying the token’s signature. Importantly, the certificate also contains a secure reference to the private key, which is used for signing and remains safely within Azure Key Vault.

Create the JWT Header

Next, you need to create the JWT header. The header specifies the signing algorithm and the type of token. You can include either the Key ID (kid) or the x5t, depending on your preference. Here are two examples:

Example Using Key ID (kid)

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "12345abcde-keyvault-id"
}

Example Using x5t (X.509 Thumbprint)

{
  "alg": "RS256",
  "typ": "JWT",
  "x5t": "F0FFEA8E53D3892859F13D1741167EEF50E90442"
}

Construct the JWT Payload

The payload is the main body of your JWT and contains the claims that describe who the token is from and what it is for. Common claims include the issuer (iss), subject (sub), audience (aud), and the expiry time (exp). Here’s an example of a JWT payload:

{
  "iss": "https://yourapp.example.com",
  "sub": "user@eax360.com",
  "aud": "https://api.eax360.com",
  "exp": 1700000000,
  "iat": 1600000000,
  "role": "admin"
}

In this example, the token indicates that it was issued by yourapp.example.com, is intended for a user with the email user@eax360.com, and is meant to be used with the API at api.eax360.com. The exp and iat (issued at) claims provide timing information, and an additional claim, such as role, can be used to specify the user’s permissions.

Create the Unsigned JWT String

Once you have your header and payload, both need to be Base64 URL encoded. After encoding, you concatenate them with a period (.) as a separator. For example:

BASE64URL(Header) + "." + BASE64URL(Payload)

If you encode the above examples, your unsigned JWT string might look something like:

eyJhbGciOiAiUlMyNTYiLCAidHlwIjogIkpXVCIsICJraWQiOiAiMTIzNDVhYmNkZS1rZXl2YXVsdC1pZCJ9.eyJpc3MiOiAiaHR0cHM6Ly95b3VyYXBwLmV4YW1wbGUuY29tIiwgInN1YiI6ICJ1c2VyQGV4YW1wbGUuY29tIiwgImF1ZCI6ICJodHRwczovL2FwaS5leGFtcGxlLmNvbSIsICJleHAiOiAxNzAwMDAwMDAwLCAiaWF0IjogMTYwMDAwMDAwMCwgInJvbGUiOiAiYWRtaW4ifQ

Note: The above string is for illustration purposes. The actual Base64 URL encoded string will vary depending on the precise content and formatting of your JSON objects.

Sign the JWT Using Azure Key Vault

Once you have created the unsigned JWT string (which is the concatenation of the Base64 URL encoded header and payload), the next step is to generate a digital signature to secure the token. This is done by sending the unsigned JWT (step 4) string to Azure Key Vault, which performs a cryptographic signing operation using the private key that remains securely stored within the vault.

How the Signing Operation Works

1. Key Vault Sign API Call
   The process typically involves calling the Sign operation on Azure Key Vault. This operation is part of the Key Vault’s cryptographic API.
2. Signing Request
   When you call the Sign operation, you send:
   • The algorithm to be used for signing (for example, RS256).
   • The unsigned JWT string (or more precisely, the hash of the unsigned JWT, depending on the API requirements).
3. Key Vault Action
   Azure Key Vault takes the provided unsigned JWT string and uses the associated private key to compute a digital signature. This operation is performed within the secure confines of the vault, ensuring that the private key is never exposed.

The Digital Signature

A digital signature is a cryptographic value that is generated by the private key and uniquely corresponds to the data being signed. In the context of JWT:

• The signature is produced using an algorithm like RS256, which applies the RSA signing algorithm in combination with SHA-256 hashing. Why is the hashing required? Well without this bit, the signature is always deterministic providing bad actors a way to use the signature for other purposes.
• The resulting digital signature is a unique string of bytes. When encoded in Base64 URL format, it forms the third part of the JWT.

Example of a Digital Signature

Suppose your unsigned JWT string is:

eyJhbGciOiAiUlMyNTYiLCAidHlwIjogIkpXVCJ9.eyJpc3MiOiAiaHR0cHM6Ly95b3VyYXBwLmV4YW1wbGUuY29tIiwgInN1YiI6ICJ1c2VyQGV4YW1wbGUuY29tIiwgImF1ZCI6ICJodHRwczovL2FwaS5leGFtcGxlLmNvbSIsICJleHAiOiAxNzAwMDAwMDAwLCAiaWF0IjogMTYwMDAwMDAwMH0

When you send this string to Azure Key Vault for signing, the vault processes it with the private key and returns a digital signature. For example, the returned signature might be (when encoded in Base64 URL format):

Q2h1bmtFeGFtcGxlU2lnbmF0dXJl...

Final JWT Assembly

After receiving the signature, you simply append it to your unsigned JWT string with a period separator:

eyJhbGciOiAiUlMyNTYiLCAidHlwIjogIkpXVCJ9.eyJpc3MiOiAiaHR0cHM6Ly95b3VyYXBwLmV4YW1wbGUuY29tIiwgInN1YiI6ICJ1c2VyQGV4YW1wbGUuY29tIiwgImF1ZCI6ICJodHRwczovL2FwaS5leGFtcGxlLmNvbSIsICJleHAiOiAxNzAwMDAwMDAwLCAiaWF0IjogMTYwMDAwMDAwMH0.Q2h1bmtFeGFtcGxlU2lnbmF0dXJl

This complete token is now a fully signed JWT that can be securely transmitted and verified by the recipient.

Finalise the JWT

Once you have the signature, encode it in Base64 URL format and append it to your unsigned JWT string using another period as a separator. The final JWT will have the structure:

BASE64URL(Header) + "." + BASE64URL(Payload) + "." + BASE64URL(Signature)

This complete token is now ready for use.

Use and Verify the Signed JWT

In this scenario the application requests the digital signature from Azure Key Vault. This desktop app is registered as an application in Microsoft Azure AD and uses the signed JWT as a client assertion. The backend is a function API (or the several applications I discussed earlier).

When the app builds the unsigned JWT, it follows the steps outlined above – creating the header (including either the kid or x5t) and the payload with the required claims. After encoding and concatenating these parts, the app sends the JWT string to Azure Key Vault. The vault uses its secure private key to sign the JWT and returns a digital signature.

Once the JWT is fully signed, the desktop app then uses it to assert its identity. It may send the signed JWT as part of an OAuth flow to request an access token from Microsoft Azure AD. Being a registered app, the desktop app is authorised to make this request.

On the backend, the function API receives the access token (or the client assertion) as part of the authentication process. The API then performs several key verifications:

• It extracts the header from the JWT and uses the included kid or x5t to retrieve the appropriate public key. This can be done via Azure ADs metadata endpoint or a trusted public repository.
• It verifies the signature with the public key to confirm that the token was indeed signed by the authorised source.
• It checks the claims in the payload – ensuring that the token has not expired, that the audience (aud) claim matches the function API’s expected value, and that the issuer (iss) is valid.

If all these checks pass, the function API accepts the token as valid and processes the request. This secure process ensures that the desktop app, by leveraging Azure Key Vault for signing and being registered in Azure AD, can safely authenticate to the backend function API that is protected by OAuth in Azure AD.

Putting it all together

At this point, we have a fully signed JWT, securely issued using Azure Key Vault, and ready to be passed between applications as a means of authentication. The process starts when an application, such as a mobile app or a backend service, needs to authenticate a request. It first constructs the JWT header and payload, including critical claims like issuer (iss), subject (sub), audience (aud), and expiry (exp). This unsigned JWT string is then sent to Azure Key Vault, where the private key—securely stored and never exposed—is used to sign it. Once the signed JWT is returned, it’s attached as a bearer token in the Authorization header of an API request.

When another service or API receives this request, it extracts the JWT and begins verification. First, it inspects the token header to identify the correct key for validation, whether by kid (Key ID) or x5t (X.509 thumbprint). It then fetches the corresponding public key, either from Azure AD’s metadata endpoint or a trusted key repository, and verifies the signature to confirm the token’s authenticity. The API also checks whether the token has expired, if the aud claim matches its expected audience, and if the issuer is valid. If everything checks out, access is granted.

This flow allows for authentication across multiple services without exposing private keys or relying on static secrets. By using Azure Key Vault for signing, we ensure that no application has direct access to the cryptographic material, reducing security risks. Additionally, JWTs eliminate the need for constant calls to an identity provider, as they can be validated locally, improving performance.

Here is a push notification scaling chart showing how authentication between platforms improves the efficiency of delivering 1 million push notifications in a day. The chart compares notifications sent without secure authentication (higher failure rates) versus with secure authentication using Azure Key Vault and JWTs (more reliable delivery). The secure authentication setup ensures that tokens are validated seamlessly, reducing failures and improving scalability. Let me know if you’d like any modifications!