You have decided you need to install Splunk to get your logs together. Or maybe someone else decided it for you and you are on the hook to make it happen. Avoid pushing through your Splunk install and ending up with it not being used. Or worse configuring it so it is not useful. You and your organization will have wasted a lot of time and money. You and your career will have missed an opportunity to shine. And you need to eventually get it done right.

Take a moment and examine some common pitfalls. Avoid these three big mistakes with your initial adoption of Splunk to improve the overall outcome, and demonstrate more value with Splunk.

1. Don’t Ignore Indexes

Indexes help you partition the logs you are indexing so that when you do searches, you can quickly tell Splunk what log sets you are interested in. Spunk has a few already built in - for the internal Splunk logs for example. And by default, Splunk uses the default index. The "default" index is convenient when you are searching because the user doesn't have to specify an index. The ease of this use might make you might think “I should just put everything in the default index to make it easy when I search.” Do not.

When you are starting to pull in data from a single application, put them in a new index. Perhaps think about the user of the logs – is it a new set of users? A new index will require the user specify the index later when doing the search. And Splunk will search only that index (instead of everything). Splunk uses indexes to optimize for performance, so searches and dashboards stay responsive by searching only your data. The bottom line is using indexes will be faster as your data grows.

With your data partitioned to indexes, you can manipulate the data in that index outside of all the other data. Let’s say you really mess something up and you need to blow away all the data - if it’s in an index, you can do it easily.

Indexes are a great way to organize the log sets that work together. Indexes along with source types allow Splunk to quickly use the sources you want and ignore the rest. If your Splunk install becomes even slightly complicated (multiple use-cases, multiple apps, etc) then you’ll be glad you put your logs into their own index.

2. Don’t Assume Splunk Knows How to Read Your Logs

Spunk knows many standard logs, and maybe the ones you are using are one of those standards. If so, that is great. However, it is likely though if you stray from the basic standard *nix or applications, you need to think about how Splunk is going to read your logs. Splunk does a great job when logs follow some basic rules - one event per line, date time stamp at the beginning and fields following the “fieldname=value” format. If you’re dealing with logs that begin to stray from some of these basic assumptions, you should beware.

If your application has a log that starts with a thread-id and continues on multiple lines like this:

123543 – 03-22-2014 03:22:01.05 Loading Plug-ins
1. Logger
2. SMTP
3. HTTP

Splunk needs to know these lines are all one event and the event date-time isn’t the first field. That means making sure you have your logs mapped to properly defined source types. Splunk uses source types to translate the log you are providing into useful events to be searched. Most importantly, Splunk needs to know two things: how to split events in the log, and what date time to assign to each event. With these two things, your events in Splunk become incredibly useful. Get these wrong, and you might find one event is divided into many with date time stamps that make no sense.

Invest the time to configure Splunk to split events properly - if they are not one event per line and use the most meaningful date-time stamp for those events. See this article on Splunk and the importance of source types.

3. Don’t Just Go It Alone

Maybe this is your project, and you need to drive it to conclusions. Probably you are the one person who is going to make sure this Splunk install gets done and gets done right. The thing to remember about Splunk is that the power of the solution comes when lots of people are using it. So do not just go it alone, recruit some colleagues to be early adopters of Splunk.

The different between a lone-nut and a leader is that a leader has followers. You don’t need to end up being a lone-nut Splunk evangelist in your organization. You will not get the fame and glory of solving the big problems if you’re the only one who knows that Splunk exists. Early on in your Splunk implementation, include some of your best folks who you think will be helped by the new Splunk install and push them to use the solution. Having a few followings increases your chances 2-3x just you alone.

In my experience, Splunk can help automate some of the most cumbersome log analysis to be quick and easy. If people still come to you, however, you’re still the single-point-of-failure. And the value of Splunk is still limited by your use of the solution. When I initially engaged some of the early adopters of Splunk in my organization, I found new ways to use Splunk. We went from basic troubleshooting to advanced performance monitoring because that was what my development team needed Splunk to analyze.

When you are ready to roll it out, make a splash. Get you and your entourage in front of folks to show off what Splunk can do. Sharing the power of the solution with what you have done and what is possible will inspire the n natural curiosity across your team and your company. Get out there with the message and be sure to recruit other power users to give you more ability to deliver results.