SPLK-1003 Questions and Answers

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles

To determine the order of directories for evaluating configuration file precedence, Splunk software considers each file's context. Configuration files operate in either a global context or in the context of the current app and user: Global. Activities like indexing take place in a global context. They are independent of any app or user. For example, configuration files that determine monitoring or indexing behavior occur outside of the app and user context and are global in nature. App/user. Some activities, like searching, take place in an app or user context. The app and user context is vital to search-time processing, where certain knowledge objects or actions might be valid only for specific users in specific apps.

The outputs.conf file defines how forwarders send data to receivers1. You can specify some output configurations at installation time (Windows universal forwarders only) or the CLI, but most advanced configuration settings require that you edit outputs.conf1.

The [tcpout:…] stanza specifies a group of forwarding targets that receive data over TCP2. You can define multiple groups with different names and settings2.

The server setting lists one or more receiving hosts for the group, separated by commas2. If you specify multiple hosts, the forwarder load balances the data across them2.

Therefore, option A is correct, because the forwarder will send data to both inputsl.mysplunkhfs.corp:9997 and inputs2.mysplunkhfs.corp:9997, even if 145.188.183.184:9097 is unreachable.

The correct answer is D. Place both props . conf and transforms . conf on the Heavy Forwarder for source A, and place both props . conf and transforms . conf on the indexer for source B.

According to the Splunk documentation1, to mask sensitive data from raw events, you need to use the SEDCMD attribute in the props.conf file and the REGEX attribute in the transforms.conf file. The SEDCMD attribute applies a sed expression to the raw data before indexing, while the REGEX attribute defines a regular expression to match the data to be masked. You need to place these files on the Splunk instance that parses the data, which is usually the indexer or the heavy forwarder2. The universal forwarder does not parse the data, so it does not need these files.

For source A, the data is routed through a heavy forwarder, which can parse the data before sending it to the indexer. Therefore, you need to place both props.conf and transforms.conf on the heavy forwarder for source A, so that the masking takes place before indexing.

For source B, the data is routed directly to the indexer, which parses and indexes the data. Therefore, you need to place both props.conf and transforms.conf on the indexer for source B, so that the masking takes place before indexing.

References: 1: Redact data from events - Splunk Documentation 2: Where do I configure my Splunk settings? - Splunk Documentation

https://docs.splunk.com/Documentation/Splunk/8.0.1/Forwarding/Aboutforwardingandreceivingdata

https://docs.splunk.com/Documentation/Forwarder/8.1.1/Forwarder/Configureforwardingwithoutputs.conf#:~:text=compressed%3Dtrue%20This%20tells%20the,the%20forwarder%20sen ds%20raw%20data.

This is the endpoint URI used to collect data using the HTTP Event Collector (HEC), which is a token-based API that allows you to send data to Splunk Enterprise from any application that can make an HTTP request. The endpoint URI consists of the protocol (http or https), the hostname or IP address of the Splunk server, the port number (default is 8088), and the service name (services/collector). For example:

https://mysplunkserver.example.com:8088/services/collector

"The search head replicates the knowledge bundle periodically in the background or when initiating a search. " "As part of the distributed search process, the search head replicates and distributes its knowledge objects to its search peers, or indexers. Knowledge objects include saved searches, event types, and other entities used in searching accorss indexes. The search head needs to distribute this material to its search peers so that they can properly execute queries on its behalf."

https://docs.splunk.com/Splexicon:Searchpeer

"A Splunk platform instance that responses to search requests from a search head. The term "Search peer" is usually synonymous with the indexer role in a distributed search topology..."

https://docs.splunk.com/Documentation/Splunk/latest/Deploy/Datapipeline "The data pipeline segments in depth. INPUT - In the input segment, Splunk software consumes data. It acquires the raw data stream from its source, breaks it into 64K blocks, and annotates each block with some metadata keys. The keys can also include values that are used internally, such as the character encoding of the data stream, and values that control later processing of the data, such as the index into which the events should be stored. PARSING Annotating individual events with metadata copied from the source-wide keys. Transforming event data and metadata according to regex transform rules."

Users log on to the search head and run reports: – The search head dispatches searches to the peers – Peers run searches in parallel and return their portion of results – The search head consolidates the individual results and prepares reports

Reference https://docs.splunk.com/Documentation/Splunk/8.2.3/Admin/Configurationparametersandthedatapipeline

 The parsing phase is the process of extracting fields and values from raw data. The parsing phase respects LINE_BREAKER, SHOULD_LINEMERGE, BREAK_ONLY_BEFORE_DATE, and all other line merging settings in props.conf. These settings determine how Splunk breaks the data into events based on certain criteria, such as timestamps or regular expressions. The event boundaries are defined by the props.conf file, which can be modified by the administrator. Therefore, the parsing phase is the last opportunity for defining event boundaries.

The possible causes of the load balancing issue on the Universal Forwarder are A and C. The receiving port and the DNS record are both factors that affect the ability of the Universal Forwarder to distribute data across multiple receivers. If the receiving port is not properly set up to listen on the right port, or if the DNS record used is not set up with a valid list of IP addresses, the Universal Forwarder might fail to connect to some or all of the receivers, resulting in poor load balancing.

https://docs.splunk.com/Documentation/Splunk/7.3.0/Admin/Wheretofindtheconfigurationfiles

"To determine the order of directories for evaluating configuration file precendence, Splunk software considers each file's context. Configuration files operate in either a global context or in the context of the current app and user"

https://docs.splunk.com/Documentation/Splunk/7.0.3/Forwarding/Routeandfilterdatad#Perform_selective_indexing_and_forwarding

Specifies a comma-separated list of tcpout group names. Use this setting to selectively forward your data to specific indexers by specifying the tcpout groups that the forwarder should use when forwarding the data. Define the tcpout group names in the outputs.conf file in [tcpout:] stanzas. The groups present in defaultGroup in [tcpout] stanza in the outputs.conf file.

https://docs.splunk.com/Documentation/Splunk/8.1.1/Updating/Definedeploymentclasses#Ways_to_define_server_classes "When you use forwarder management to create a new server class, it saves the server class definition in a copy of serverclass.conf under $SPLUNK_HOME/etc/system/local. If, instead of using forwarder management, you decide to directly edit serverclass.conf, it is recommended that you create the serverclass.conf file in that same directory, $SPLUNK_HOME/etc/system/local."

https://docs.splunk.com/Documentation/Splunk/8.0.3/DistSearch/Distributedsearchgroups

https://answers.splunk.com/answers/131127/scripted-authentication.html

Scripted Authentication: An option for Splunk Enterprise authentication. You can use an authentication system that you have in place (such as PAM or RADIUS) by configuring authentication.conf to use a script instead of using LDAP or Splunk Enterprise default authentication.

https://docs.splunk.com/Documentation/Splunk/8.0.6/Security/ManageSplunkuserroleswithLDAP

https://docs.splunk.com/Documentation/Splunk/8.2.2/DistSearch/Whatisdistributedsearch

Parallel reduce search processing If you struggle with extremely large high-cardinality searches, you might be able to apply parallel reduce processing to them to help them complete faster. You must have a distributed search environment to use parallel reduce search processing.

The indexing pipeline and the parsing pipeline are the two pipelines that are responsible for transforming the raw data into events and preparing them for indexing. The indexing pipeline applies index-time settings, such as timestamp extraction, line breaking, host extraction, and source type recognition. The parsing pipeline applies parsing settings, such as field extraction, event segmentation, and event annotation.

https://docs.splunk.com/Documentation/Splunk/latest/Data/Whitelistorblacklistspecificincomingdata#Include_or_exclude_specific_incoming_data

https://docs.splunk.com/Documentation/Forwarder/8.0.5/Forwarder/Configuretheuniversalforwarder

--Key configuration files are: inputs.conf controls how the forwarder collects data. outputs.conf controls how the forwarder sends data to an indexer or other forwarder server.conf for connection and performance tuning deploymentclient.conf for connecting to a deployment server

Hot/warm/cold/thawed bucket types are searchable. Frozen isn't searchable because its either deleted at that state or archived.

https://docs.splunk.com/Documentation/Splunk/8.0.5/Data/Usingforwardingagents

Scroll down to the section Titled, How to configure forwarder inputs, and subsection Here are the main ways that you can configure data inputs on a forwarder Install the app or add-on that contains the inputs you wants

Search-time field extractions are the process of extracting fields from events after they are indexed. Search-time field extractions are specified on the search head, which is the Splunk component that handles searching and reporting. Search-time field extractions are configured in props.conf and transforms.conf files, which are located in the etc/system/local directory on the search head. Therefore, option D is the correct answer. References: Splunk Enterprise Certified Admin | Splunk, [About fields - Splunk Documentation]

According to the Splunk documentation1, to manually specify a character set for an input, you need to set the CHARSET key in the props.conf file. You can specify the character set by host, source, or sourcetype, but not by index.

https://docs.splunk.com/Documentation/Splunk/latest/Data/Configurecharactersetencoding

Adding further clarity and quoting same Splunk reference URL from @giubal"

"To keep configuration settings consistent across peer nodes, configuration files are managed from the cluster master, which pushes the files to the slave-app directories on the peer nodes. Files in the slave-app directories have the highest precedence in a cluster peer's configuration. Here is the expanded precedence order for cluster peers:

1.Slave-app local directories -- highest priority

2. System local directory

3. App local directories

4. Slave-app default directories

5. App default directories

6. System default directory --lowest priority

Using the Splunk reference URL https://docs.splunk.com/Splexicon:Searchpeer

"search peer is a splunk platform instance that responds to search requests from a search head. The term "search peer" is usally synonymous with the indexer role in a distributed search topology. However, other instance types also have access to indexed data, particularly internal diagnostic data, and thus function as search peers when they respond to search requests for that data."

URL https://docs.splunk.com/Documentation/Splunk/8.2.2/DistSearch/Configuredistributedsearch

"To activate distributed search, you add search peers, or indexers, to a Splunk Enterprise instance that you desingate as a search head. You do this by specifying each search peer manually."

This option corresponds to the file path “$SPLUNK_HOME/etc/apps/splunk_TA_nginx/local/inputs.conf”. This is the configuration file that the user needs to edit to ingest the NGINX access logs to ensure it remains unaffected after upgrade. This is explained in the Splunk documentation, which states:

The local directory is where you place your customized configuration files. The local directory is empty when you install Splunk Enterprise. You create it when you need to override or add to the default settings in a configuration file. The local directory is never overwritten during an upgrade.

Immediately after installation, a universal forwarder will start generating internal Splunk logs that contain information about its own operation, such as configuration changes, data inputs, and forwarding activities1. These logs are stored in the $SPLUNK_HOME/var/log/splunk directory on the universal forwarder machine1. The universal forwarder will not automatically detect any indexers in its subnet and begin routing data, as it needs to be configured with the IP address and port number of the indexer or the deployment server2. The universal forwarder will not begin reading local files on its server, as it needs to be configured with the data inputs that specify which files or directories to monitor2. The universal forwarder will not send an email to the operator that the installation process has completed, as this is not a default behavior of the universal forwarder and would require additional configuration3.

https://docs.splunk.com/Documentation/Splunk/8.1.1/Indexer/HowSplunkstoresindexes

Once further conditions are met (for example, the index reaches some maximum number of warm buckets), the indexer begins to roll the warm buckets to cold, based on their age. It always selects the oldest warm bucket to roll to cold. Buckets continue to roll to cold as they age in this manner. Cold buckets reside in a different location from hot and warm buckets. You can configure the location so that cold buckets reside on cheaper storage.

http://www.splunk.com/view/SP-CAAAGPR

The app local directories move to second in the priority list. This is explained in the Splunk documentation, which states:

In a clustered environment, the precedence of configuration files changes slightly from that of a standalone deployment. The app local directories move to second in the priority list, after the peer-apps local directory. This means that any configuration files in the app local directories on the individual peers are overridden by configuration files of the same name and type in the peer-apps local directory on the master node.

After an app is downloaded, it resides under $SPLUNK_HOME/etc/apps on the deployment clients. But it resided in the $SPLUNK_HOME/etc/deployment-apps location in the deployment server.

The failOpen setting in the [duoMFA] stanza determines how Splunk software handles authentication requests when it cannot connect to the Duo Security service. If failOpen is set to True, as in this example, Splunk software allows users to log in without completing a multifactor challenge. If failOpen is set to False, Splunk software denies all logins when it cannot connect to Duo Security. This setting is independent of the authentication type or the secretKey protection. References = Connect to Duo Security for multifactor authentication

The coldPath parameter defines the path for the cold buckets, which are the oldest and least frequently accessed data in an index1. By setting the coldPath to point to the NAS mount point, the Splunk administrator can achieve the retention strategy of having older data on slower NAS storage.

According to the Splunk documentation1, to customize a configuration file, you need to create a new file with the same name in a local or app directory. Then, add the specific settings that you want to customize to the local configuration file. Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. The Splunk Enterprise upgrade process overwrites the default directory.

To deploy configuration files to deployment clients, you need to use the deployment server. The deployment server is a Splunk Enterprise instance that distributes content and updates to deployment clients2. The deployment server uses a directory called $SPLUNK_HOME/etc/deployment-apps to store the apps and configuration files that it deploys to clients2. To update the configuration files in this directory, you need to edit them manually and then run the command $SPLUNK_HOME/bin/sp1unk reload deploy—server to make the changes take effect2.

Therefore, option A is incorrect because it does not include the reload command. Option B is incorrect because it makes the change on a deployment client instead of the deployment server. Option D is incorrect because it changes the default directory instead of the local directory.

References: 1: How to edit a configuration file - Splunk Documentation 2: Deployment of configuration files - Splunk Community

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Protectagainstlossofin-flightdata#:~:text=The%20default%20for%20the%20maxQueueSize,wait%20queue%20size%20is%2021MB .

https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Protectagainstlossofin-flightdata

https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/Aboutlicenseviolations

"Enterprise Trial license. If you get five or more warnings in a rolling 30 days period, you are in violation of your license. Dev/Test license. If you generate five or more warnings in a rolling 30-day period, you are in violation of your license. Developer license. If you generate five or more warnings in a rolling 30-day period, you are in violation of your license. BUT for Free license. If you get three or more warnings in a rolling 30 days period, you are in violation of your license."

Begin generating internal Splunk logs. Immediately after installation, a Universal Forwarder will start generating internal Splunk logs that contain information about its own operation, such as startup and shutdown events, configuration changes, data ingestion, and forwarding activities1. These logs are stored in the $SPLUNK_HOME/var/log/splunk directory on the Universal Forwarder machine2.