Splunk and the self-signed certificate on port 8089

I'm writing this post after finding a solution to this problem. Pieces of the solution were scattered around the web but I didn't find them all in one place.

Problem

Splunk's ports when accessed using SSL/TLS are by default protected with a self-signed certificate. Many Enterprises are beginning to scan for these cases and flagging them for remediation so that the encrypted communications are protected by a certificate signed by the Enterprise itself.

Using an alternate certificate for the Splunk web UI (port 8000 by default) is well-documented but I did not feel that it was documented well for the management port (port 8089 by default).

Solution

The solution has a few steps:

1. Generate a Certificate Signing Request (CSR) and private key.
2. Use the CSR to obtain a signed certificate from a Certificate Authority (CA)
3. Obtain the Root CA certificate chain for the organization that provided the signed certificate
4. Combined outputs of steps 1-3 as required by Splunk
5. Configure Splunk to use the items in step 4
6. Restart Splunk

Before going further, consider whether you need the management port to be enabled for Universal Forwarders (UF). It is not required for forwarder management from the web UI, nor for deployment apps. It is required for API or CLI communication with the UF. If you don't use these features then you can simple disable the port by putting the following in server.conf and restarting the UF.

[httpServer]
disableDefaultPort = true

However, if you want to leave the port open and protect it with your own certificate then read on.

And, unless you have changed the default configuration, Splunk KV stores on the same server will also be protected by the configuration applied in this post.

Step 1: Generate a Certificate Signing Request (CSR) and private key.

These steps will leave you with a CSR stored in server_conf.csr and a private key in server_conf.key

Linux

openssl req -out server_conf.csr -new -newkey rsa:2048 -keyout server_conf.key

Windows

REM SPLUNK_HOME is the root of your Splunk Enterprise installation set SPLUNK_HOME="C:\Program Files\Splunk"

REM TMP will hold the generated private key and CSR files

set TMP=C:\TEMP REM Generate the private key for the certificate.

%SPLUNK_HOME%\bin\splunk cmd openssl genrsa -des3 -out %TMP%\server_conf.key 2048

REM Generate the CSR request file

%SPLUNK_HOME%\bin\splunk cmd openssl req -new -key %TMP%\server_conf.key -out %TMP% \server_conf.csr

You should leave this step with two outputs:

• CSR file
• Private key

Step 2: Use the CSR to obtain a signed certificate from a Certificate Authority (CA)

Step 3: Obtain the Root CA certificate chain for the organization that provided the signed certificate

The method to accomplish Step 2 and 3 will vary by CA, but you will normally need to provide your CSR file as part of the process.

You should leave these steps with:

• CA-signed certificate provided by your CA
• Root CA and Intermediate CA certificates provided by your CA

Step 4: Combine outputs of steps 1-3 as required by Splunk

All of the files you have created so far are plaintext files. They need to be combined in specific ways:

• Root CA and Intermediate CA certificates combined into a single file (example: server_conf_root.pem)
• CA-signed certificate and private key (example: server_conf.pem)

By "combined", I literally mean to copy and paste the contents of the files you received into a single file, one after the other. The example filenames above will be used in subsequent steps.

Store the files in a location accessible by your Splunk installation that will not be affected by upgrades. For example, you may choose to create a directory like $SPLUNK_HOME/etc/auth/mycerts, giving you these files:

• $SPLUNK_HOME/etc/auth/mycerts/server_conf_root.pem
• $SPLUNK_HOME/etc/auth/mycerts/server_conf.pem

Step 5: Configure Splunk to use the items in step 4

Modify your server.conf file to include these attributes:

[sslConfig]

enableSplunkdSSL = true

serverCert = /opt/splunk/etc/auth/mycerts/server_conf.pem

sslRootCAPath = /opt/splunk/etc/auth/mycerts/server_conf_root.pem

sslPassword = <key password entered during CSR creation>

Note that, when you restart Splunk in a subsequent step, the sslPassword value will be replaced with a hash of the value by Splunk. As long as everything is working you do not need to worry about it.

Step 6: Restart Splunk

This step hopefully does not need any elaboration!

After the restart, you can use a browser to access the management port (i.e. https://splunk.mycompany.com:8089) and confirm that it is using your CA-signed certificate using the browser's certificate inspection functionality.

Unless you have changed the default configuration, Splunk KV stores on the same server will also be protected by the configuration applied in this post.