All about time

Time is an important and confusing topic in Splunk. If you want to skip this section, absorb one concept—time must be parsed properly on the way into the index as it cannot be changed later without indexing the raw data again.

How Splunk parses time

If given the date 11-03-04, how would you interpret this date? Your answer probably depends on where you live. In the United States, you would probably read this as November 3, 2004. In Europe, you would probably read this as March 11, 2004. It would also be reasonable to read this as March 4, 2011.

Luckily, most dates are not this ambiguous, and Splunk makes a good effort to find and extract them, but it is absolutely worth the trouble to give Splunk a little help by configuring the time format. We'll discuss the relevant configurations in Chapter 11, Configuring Splunk.

How Splunk stores time

Once the date is parsed, the date stored in Splunk is always stored as GMT epoch. Epoch time is the number of seconds since January 1, 1970. By storing all events using a single time zone, there is never a problem lining up events that happen in different time zones. This, of course, only works properly if the time zone of the event can be determined when it is indexed. This numeric value is stored in the field _time.

How Splunk displays time

The text of the original event, and the date it contains, is never modified. It is always displayed as it was received. The date displayed to the left of the event is determined by the time zone of the Splunk instance or the user's preference, as specified in Your account.

How time zones are determined and why it matters

Since all events are stored according to their GMT time, the time zone of an event only matters at parse time, but it is vital to get it right. Once the event is written into the index, it cannot be changed without reindexing the raw data.

The time zone can come from a number of places, in the following order of precedence:

• The time zone specified in the log. For instance, the date 2012-02-07T01:03:23.575-0600, -0600 indicates that the zone is 6 hours behind GMT. Likewise, Tue 02 Feb, 01:03:23 CST 2012 represents the same date.
• The configuration associated with a source, host, or sourcetype, in that order. This is specified in props.conf. This can actually be used to override the time zone listed in the log itself, if needed. We will discuss this in Chapter 11, Configuring Splunk.
• The time zone of the Splunk instance forwarding the events. The time zone is relayed along with the events, just in case it is not specified elsewhere. This is usually an acceptable default. The exception is when different logs are written with different time zones on the same host, without the time zone in the logs. In that case, it needs to be specified in props.conf.
• The time zone of the Splunk instance parsing the events. This is sometimes acceptable and can be used in interesting ways in distributed environments.
• The important takeaway, again, is that the time zone needs to be known at the time of parsing and indexing the event.

Different ways to search against time

Now that we have our time indexed properly, how do we search against time? The Date & Time Range picker provides a neat set of options for dealing with search times:

This picker widget is organized by:

• Presets
• Relative
• Real-time
• Data Range
• Date & Time Range and
• Advanced

Let's take a look at understanding each of these.

Presets

Presets are time ranges that are pre-defined for you in Splunk Enterprise. You should be aware, though, that if you are searching potentially large amounts of data, results will return faster if you run the search over a smaller time period (rather than All time).

Relative

If the Relative presets are not what you need, you can use the custom Relative time range options to specify a time range for your search that is relative to now. You can select from the list of time range units, Seconds Ago, Minutes Ago, and so on:

Splunk also provides the ability to use Beginning of second (the default) or a No Snap-to time unit to indicate the nearest or latest time to which your time amount rounds up to. If you don't specify a snap to time unit, Splunk snaps automatically to the second. Unlike the Presets, to actually apply your (Relative) selections to the search, you need to click the Apply button.

Real-time

The custom Real-time option gives you the ability to set the start time for your real-time time range window. Keep in mind that the search time ranges for historical searches are set at the time at which the search runs. With real-time searches, the time ranges are constantly updating and the results accumulate from the beginning of your search.

You can also specify a time range that represents a sliding window of data, for example, the last 30 seconds.

When you specify a sliding window, Splunk takes that amount of time to accumulate data. For example, if your sliding window is 5 minutes, you will not start to see data until after the first 5 minutes have passed.

Windowed real-time versus all-time real-time searches

When designing your searches, it's important to keep in mind that there is a difference between Splunk real-time searches that take place within a set window (like 30 seconds or 1 minute) and real-time searches that are set to All time.

In windowed real-time searches, the events in the search can disappear as they fall outside of the window, and events that are newer than the time the search job was created can appear in the window when they occur.

In all-time real-time searches, the window spans all of your events, so events do not disappear once they appear in the window. But events that are newer than the time the search job was created, can appear in the window as they occur.

In comparison, in historical searches, events never disappear from within the set range of time that you are searching and the latest event is always earlier than the job creation time (with the exception of searches that include events that have future-dated timestamps).

Date range

You can use the custom Date Range option to add calendar dates to your search. You can choose among options to return events: Between a beginning and end date, Before a date, and Since a date (for these fields, you can either type the date into the text box, or select the date from a calendar).

Date and time range

Use the custom Date & Time Range option to specify calendar dates and times for the beginning and ending of your search. Again, you can type the date into the text box or select the date from a calendar.

Advanced

Use the Advanced option to specify the earliest and latest search times. You can write the times in Unix (epoch) time or relative time notation. The epoch time value that you enter is converted to local time. This timestamp is displayed under the text field so that you can verify your entry.

Specifying time in-line in your search

You can also directly use relative and exact times in your searches. For instance, given the search item bob error, you can specify the time frame you want to use directly in the search, using the fields earliest and latest.

• To search for errors affecting bob in the last 60 minutes, use earliest=-60m bob error
• To search for errors affecting bob in the last 3 hours, snap to the beginning of the hour using earliest=-3h@h bob error
• To search for errors affecting bob yesterday, use earliest=-1d@d latest=-0d@d bob error
• To search for errors affecting bob since Monday midnight, use earliest=-0@w1 bob error

You cannot use different time ranges in the same query; for instance, in a Boolean search, (earliest=-1d@d latest=-0d@d bob error) OR (earliest=-2d@d latest=-1d@d mary error) will not work. The append command provides a way of accomplishing this.

_indextime versus _time

It is important to note that events are generally not received at the same time as stated in the event. In most installations, the discrepancy is usually of a few seconds, but if logs arrive in batches, the latency can be much larger. The time at which an event is actually written in the Splunk index is kept in the internal field _indextime.

The time that is parsed out of the event is stored in _time.

You will probably never search against _indextime, but you should understand that the time you are searching against is the time parsed from the event, not the time at which the event was indexed.