Ever wanted to build a complex Splunk environment for testing, which looks as close as possible to a production deployment? Need to test a Splunk upgrade? See how Splunk indexer- or search head clustering works? Or just need to verify some configuration changes? This is the right place for you! The aim of this framework is to produce a Splunk environment in a fast and convenient way for testing purposes or maybe also for production use. The created Splunk installation and setup follows best practices. There are many ways to configure a Splunk environment in terms of configuration file locations, so this is just another example how to do it.
- Splunk Platform Automator
- Table of Contents
- Support
- Features
- Roadmap
- Changelog
- Installation
- Upgrade
- Removed Biased Language
- Building Windows Virtual Machine Template
- Framework Usage
- Known issues, limitations
- License
Note: This framework is not officially supported by Splunk. I am developing this on best effort in my spare time.
- Build complex, reproducible Splunk environments in one shot, including all roles available for Splunk Enterprise.
- Building Cluster Manager, Indexer Clusters, Deployer, Search Head Clusters, Deployment Server, Universal Forwarders, Heavy Forwarders, License Manager and Monitoring Console. All ready to use.
- Configuration done according best practices with configuration apps
- Splunk environment definition stored in one simple yaml file
- Example configuration files for different setups included
- Deployment and configuration done with Ansible
- AWS Infrastructure provisioning via Terraform (recommended for AWS deployments)
- Ansible-driven Terraform workflow with single config file
- Automatic inventory generation
- Support for complex configurations (volumes, instance types, etc.)
- Virtual hosts can be created by Vagrant
- Currently supports Virtualbox or AWS Cloud (legacy).
- Can deploy Splunk on existing hosts (virtual or physical)
- Developed and tested on MacOSX but should support Linux as well.
See the upcoming features in the Roadmap
Implemented changes are to be found in the Changelog
The Framework is currently tested on Mac OSX and Linux, but any other Unix, which is supported by Virtualbox, should work too.
- Make sure you have Python 3.9+ installed. If not available for your distro, you can build your own Python version.
- (Optional) Download and install Vagrant. If you are using the VirtualBox plugin.
- Install Ansible, I personally prefer Brew (on OSX) which makes it as easy as
brew install ansible. For supported Ansible versions check here - Create a folder called
Vagrantand change into it. - Download and extract a Splunk Platform Automator release here or clone from GitHub when using the master branch:
git clone https://github.com/splunk/splunk-platform-automator.git - Create a folder called
Software. - Download the tgz. archive for the Splunk Software and put in the
Softwaredirectory - Copy Splunk Professional Services Best Practices Base Config Apps and extract them into the
Softwaredirectory. The Apps are not available for public download, please contact your Splunk Professional Services representative to get them.- Configurations Base Apps
- Configurations Cluster Apps
- If you have a Splunk License file, link it to the name
Splunk_Enterprise.licinside theSoftwaredirectory.
Your directory structure should now look like this:
./Vagrant/splunk-platform-automator/...
./Vagrant/Software/Configurations - Base/...
./Vagrant/Software/Configurations - Index Replication/...
./Vagrant/Software/splunk-8.1.2-545206cc9f70-Linux-x86_64.tgz
./Vagrant/Software/splunkforwarder-8.1.2-545206cc9f70-Linux-x86_64.tgz
./Vagrant/Software/Splunk_Enterprise.lic- Download and install Virtualbox.
- Install the Virtualbox plugin for Vagrant:
vagrant plugin install vagrant-vbguest
The Windows Subsystem for Linux does allow to run Splunk Platform Automator on Windows. It even allows to create virtualbox hosts from it on the windows host directly.
Execute the steps above. To allow vagrant to talk to virtualbox follow the steps below.
- Create /etc/wsl.conf and reboot WSL (
wsl --shutdown)
[automount]
options = "metadata"- Enable WSL2 port forwarding by installing a vagrant plugin with:
vagrant plugin install virtualbox_WSL2 - Add Environment Variables in WSL (maybe to your
~/.bashrc)
export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"
export PATH="$PATH:/mnt/c/Program Files/Oracle/VirtualBox"
⚠️ Note: This section describes the legacy Vagrant AWS plugin setup. For new AWS deployments, we recommend using the Terraform approach instead, which is more modern and easier to manage.
- Install either of the aws vagrant plugins:
- vagrant-aws: This is te orig plugin but not maintained anymore and has issues with newer vagrant versions on OSX. The last working version of vagrant is 2.3.4. Install it with
vagrant plugin install vagrant-aws - vagrant-gecko-aws: This is a clone of the orig project and does support newer versions (up to 2.3.7) of vagrant. Install it with
vagrant plugin install vagrant-gecko-aws --entry-point vagrant-aws
- vagrant-aws: This is te orig plugin but not maintained anymore and has issues with newer vagrant versions on OSX. The last working version of vagrant is 2.3.4. Install it with
- Download the vagrant dummy box for aws:
vagrant box add aws-dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box - Generate AWS ACCESS Keys, described here
- Optional, but recommended:
- Add AWS_ACCESS_KEY_ID=<your access key ID> as environment variable
- Add AWS_SECRET_ACCESS_KEY=<your secret access key> as environment variable
- Create an ssh key pair described here and store the public key on your disk for later reference in the config file
- Create an AWS security group and name it for example 'Splunk_Basic' and add the following TCP ports
| Type | Protocol | Port Range | Source | Description |
|---|---|---|---|---|
| All TCP | TCP | 0 - 65535 | 172.31.0.0/16 | Allow all internal traffic |
| Custom TCP | TCP | 8000 | 0.0.0.0/0 | Splunk Web Interface |
| SSH | TCP | 22 | 0.0.0.0/0 | SSH to all hosts |
| Type | Protocol | Port Range | Destination | Description |
|---|---|---|---|---|
| All Traffic | All | All | 0.0.0.0/0 | Allow all traffic |
To upgrade your Splunk Platform Automator, just update your local code from the repo
git pullPlease refer to the Migration Guide.
Please refer to the Removed Biased Language Guide.
To build your own windows vagrant image follow Setup Windows Vagrant image
Run vagrant the first time to initialize itself and create needed directories. You must execute vagrant always in side the Splunk Platform Automator directory where the Vagrantfile sits, otherwise it will not work correctly. You will see the usage page, when executing vagrant without options.
cd splunk-platform-automator
vagrantThere is one single configuration file, where all settings for your deployment are defined. Copy one configuration file from the examples to config/splunk_config.yml and adjust the setting to your needs. For a standard setup you should be fine with most of the default settings, but there are a lot of things you can adjust for special cases. See the configuration description file, where all existing values are described.
AWS: See instruction here when deploying into Amazon Cloud. You can start with splunk_config_terraform_aws.yml for a simple environment. Copy splunk_idxclusters, splunk_shclusters and splunk_hosts sections from other examples for more complex deployments.
Splunk Platform Automator supports multiple deployment targets. Choose the appropriate method for your environment:
When building virtual machines for Virtualbox the first time it will pull an OS image from the internet. The box images are cached here: ~/.vagrant.d/boxes.
Create the Virtual Machines:
vagrant upRun Ansible playbooks to deploy and configure the Splunk software:
The vagrant up command only creates the virtual machines. To deploy Splunk afterwards, run this command:
ansible-playbook ansible/deploy_site.ymlTo run both steps with one command use:
vagrant up; ansible-playbook ansible/deploy_site.ymlModern, declarative infrastructure provisioning using Terraform managed by Ansible playbooks.
Prerequisites:
- Terraform 1.3.0+ installed
- AWS CLI installed (required for instance status checks)
community.generalAnsible collection:ansible-galaxy collection install community.general- AWS credentials (via environment variables or config file)
- AWS security group created (e.g., 'Splunk_Basic') - see security group example
- EC2 key pair created
Quick Start:
- Configure
config/splunk_config.ymlwith aterraform.awssection:
terraform:
aws:
region: "eu-central-1"
ami_id: "ami-03cbad7144aeda3eb" # Redhat 9
key_name: "aws_key"
ssh_private_key_file: "~/.ssh/aws_key.pem"
ssh_username: "ec2-user"
security_group_names: ["Splunk_Basic"]
instance_type: "t2.micro"
splunk_hosts:
- name: idx1
roles: [indexer]
terraform:
aws:
instance_type: "c5.4xlarge"
root_volume_size: 100- Provision infrastructure:
ansible-playbook ansible/provision_terraform_aws.yml- Deploy Splunk:
ansible-playbook ansible/deploy_site.yml- Destroy infrastructure:
ansible-playbook ansible/destroy_terraform_aws.ymlFeatures:
- ✅ Single source of truth in
splunk_config.yml - ✅ Automatic Ansible inventory generation
- ✅ Support for
iterto generate multiple hosts with numbering - ✅ Support for
listto generate multiple hosts with custom names - ✅ Per-host instance types, volumes, and configurations
- ✅ AWS credentials can be in config or environment variables
Documentation:
- Ansible-Terraform Integration Guide - Complete documentation
- Terraform AWS README - Terraform configuration details
Traditional Vagrant-based approach using the vagrant-aws plugin.
⚠️ Note: This method is considered legacy. The Terraform approach (Option B) above is recommended for new AWS deployments.
To use the Vagrant AWS plugin:
- Follow the AWS plugin installation instructions
- Configure
config/splunk_config.ymlwith anawssection (see splunk_config_aws.yml) - Run
vagrant upto create instances - Run
ansible-playbook ansible/deploy_site.ymlto deploy Splunk
This will gracefully shutdown all the virtual machines.
vagrant haltYou can destroy all the virtual machines with one command.
vagrant destroy [-f] [<hostname>]Ansible playbooks can be run over and over again. If the virtual machine is already built, you can rerun the playbooks on a certain host again. This can be needed if something fails and you fixed the error.
ansible-playbook ansible/deploy_site.yml [--limit <hostname>]To login to one of the hosts just open the index.html file created in the splunk-platform-automator/config directory. You will find links to every role of your deployment.
If something changes along the way and you need to update the linkpage just call this playbook:
ansible-playbook ansible/create_linkpage.ymlspash is a helper script that makes SSH connections easier by looking up host details directly from the Ansible inventory. It handles keys, users, and IP addresses automatically.
Usage:
# SSH into a host (matches partial names)
./bin/spash <hostname>
# List all available hosts
./bin/spash -l
# Pass extra arguments to SSH
./bin/spash idx1 -L 8089:localhost:8089Vagrant deployes an ssh key for the vagrant user to login without a password.
vagrant ssh <hostname>Vagrant uses a dedicated user to work inside the virtual machines. The user name is vagrant and has sudo rights to switch to root or other users.
Splunk Enterprise is installed and run as user splunk. You can switch to this user by sudo su - splunk. For convenience, I have added some command aliases to the user vagrant and user splunk.
aliasspash can also be used to copy files to and from hosts using the scp protocol. It automatically resolves the connection details from the inventory.
# Copy a local file to a remote host
./bin/spash -c local_file.txt idx1:/tmp/
# Copy a remote file to the current directory
./bin/spash -c idx1:/opt/splunk/etc/system/local/server.conf .You can copy files from your host system to the virtual nodes with the vagrant command. You need to install the vagrant plugin vagrant-scp to have this feature available. Check Vagrant Docs on how to do this.
vagrant scp <files> <target_on_dest> [vm_name]vagrant scp ../app_dir/splunk-add-on-for-unix-and-linux_831.tgz /var/tmp ufYou can also use the ansible playbooks without vagrant. For that you have to create your virtual or physical machines by other means. You can use the ansible playbooks to
deploy the Splunk roles onto the existing servers. Specify the hostnames in the splunk_config.yml file in the splunk_hosts section.
Ansible needs to know where to connect to via ssh to run the playbooks. For this you need to create some custom variables in the
splunk_config.yml file.
As a minimum specify the ssh user for ansible and the ssh private key which has been deployed on the systems. This user must be able to elevate to the root user with sudo.
custom:
ansible_user: ansible
ansible_ssh_private_key_file: '~/.ssh/id_rsa'If you have host specific variables the custom section can also be added on host level. This could be for example ansible_host if different from the hostname. Also check configuration description
You can verify things like this first with an ansible ping:
ansible -m ping allAnd then some more ansible prerequisites with this playbook
ansible-playbook ansible/test_ansible_prereqs.ymlYou can build your own python version, if you need a specific python version or your local one is outdated. Please install the needed development tools in order to be able to compile stuff.
An easy way to install new python versions is using pyenv but you can also manually install Python like for example:
cd
mkdir tmp
mkdir -p ~/Python/3.9.9
cd tmp
wget https://www.python.org/ftp/python/3.9.9/Python-3.9.9.tgz
tar -xzf Python-3.9.9.tgz
cd Python-3.9.9/
./configure --prefix=~/Python/3.9.9 --with-ensurepip=install
make
make installIf you need a specific Ansible version you can create it inside a virtualenv environment. This can be useful when deploying older linux images, which too old python versions. An easy way to install new virtual environments is using pyenv-virtualenv or you can do it manually like the following example.
python3 -m venv ansible_414
source ansible_414/bin/activate
python -m pip install ansible==7.7.0This installs ansible 2.14.10, see the version mapping
You must install some additional python modules for Splunk Platform Automator to work:
- jmespath # required for json_query calls
- lxml # required for license file checks
- boto3 # required for ec2 (aws) plugin
Use the requirements file for easy installation
python -m pip install -r requirements.txtCheck the ansible version
ansible --versionIf the version is not correct, open a new terminal and activate the virtual environment again with the command from above.
source ansible_414/bin/activate- Ubuntu 20.04 fails on checking the systemd settings and needs a more current ansible version. See service_facts broken in Ubuntu 20.04
- ulimit settings not working on Ubuntu 14 (without systemd)
- Forwarding data from a universal forwarder to a heavy forwarder cannot be configured in the config file. This must be done manually after installation.
- Virtualbox: Virtual host startup does not respond sometimes, if it fails, recreate the host again.
- Virtualbox has some issues with clock time skew, when not using virtualbox additions. I added a workaround with forcing time clock sync every 5 minutes. A working internet connection on the Virtualbox host is needed.
- AWS: Due to security reasons the login page with the admin password information has been disabled and https is enabled with splunk's own self signed certs.
- AWS: OS images (AMI) do not have ntp configured by default. This will be added in Splunk Platform Automator later.
The following Ansible versions are tested and working with Splunk Platform Automator.
- ❌ Ansible 2.9.x and below (EOL)
- ✅ Ansible 2.10.x - 2.17.x (EOL)
- ✅ Ansible 2.18.x - 2.20.x
Check the Ansible Support Matrix for the most current information.
Copyright 2022 Splunk Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.