|
1 | | -Puppet Report Viewer |
2 | | -============== |
3 | | - |
4 | | -Description |
5 | | ------------ |
6 | | -This is a Splunk Addon that provides views into the status of the Puppet installation that is sending its reports to Splunk via the HEC. To use this viewer it has to be installed alongside the `splunk_hec` report processor provided in the [Puppet Forge](https://forge.puppet.com/puppetlabs/splunk_hec). The report processor sends data from Puppet to Splunk via the [HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector). |
7 | | - |
8 | | -The steps to get this addon working are: |
9 | | - |
10 | | -1. Install the Puppet Report Viewer addon |
11 | | -2. Create atleast one HEC input (puppet:summary) |
12 | | -3. Install `splunk_hec` module in Puppet environment and configure with the HEC token and Splunk Server |
13 | | -4. For the main page to update with inventory after version 2.0, you will also need to send puppet:facts information (inventory) to Splunk, see the Splunk_hec documentation on configuring fact support |
14 | | - |
15 | | -Once configured, the overview page will start showing Puppet run report status, and information about changes over various windows of time. The views can be customized, updated, modified to suit your needs. |
16 | | - |
17 | | - |
18 | | - |
19 | | -For detailed report generation, you will need to now install and configure the [Puppet Alert Actions Add-on](https://github.com/puppetlabs/TA-puppet-alert-actions) |
20 | | - |
21 | | -Advanced Configuration |
22 | | ----------------- |
23 | | -All report views support using custom indicies for storing event data. They accomplish this with a series of advanced search macros. The queries assume each sourcetype can be stored in it's own index (facts, summary reports, detailed reports, bolt events, action events, jobs, activities_rbac, activities_classifier, and Puppet Enterprise metrics). |
24 | | - |
25 | | -There is one top level macro, `puppet_index` which defaults to "", if you configure the HEC to use a different index and want all Puppet in that index, change that value here to be `index=puppetindexname`. |
26 | | - |
27 | | -If you are using [puppetlabs/splunk_hec](https://forge.puppet.com/puppetlabs/splunk_hec/readme) version 0.5.0 or later, you can specify different HEC tokens for Summary Reports, Facts, and Metrics. Then create an index and an associated HEC token associated with those sourcetypes, and configure both the splunk_hec module in Puppet with those new values. Actions, Bolt Events, and Detailed Reports are all submitted via different tools and would need ot be changed according to use a different HEC token. Then the corresponding macro's updated to use those indexes. |
28 | | - |
29 | | -For example, if you want most Puppet data to go to one index, but Facts, Metrics, and Detailed Reports to go to their own indexes, one would follow these steps: |
30 | | -- Create four indexes: puppet_data, puppet_facts_data, puppet_metrics_data, and puppet_detailed_data (or whatever name makes sense), each with their desired timespan, retention, etc. |
31 | | -- Create four HEC's (example names): |
32 | | -1. `puppet` with sourcetype of `puppet:summary` and the index `puppet_data` |
33 | | -2. `puppet_facts` with sourcetype of `puppet:facts` and the index of `puppet_facts_data` |
34 | | -3. `puppet_metrics` with sourcetype of `puppet:metrics` and the index of `puppet_metrics_data` |
35 | | -4. `puppet_detailed` with sourcetype of `puppet:detailed` and the index of `puppet_detailed_data` |
36 | | -- Configure the `splunk_hec` module with the corresponding tokens |
37 | | -1. `splunk_hec::token` with the value from the `puppet` HEC (since you want all Puppet using splunk_hec plugin to go here, except for facts and metrics) |
38 | | -2. `splunk_hec::token_facts` with the value from the `puppet_facts` HEC |
39 | | -3. `splunk_hec::token_metrics` with the value from the `puppet_metrics` HEC |
40 | | -- Update the Puppet Report Viewer's configuration to use the `puppet_detailed` HEC token, because detailed reports are pulled from Puppet and generated by the alert action in this application |
41 | | -- Update the advanced search macros to use the new values: |
42 | | -1. Open Advanced Search under the Settings -> Knowledge menu |
43 | | -2. Select `Search Macros` |
44 | | -3. Select `puppet_index` and change the definition to `index=puppet_data`, click save |
45 | | -4. Select `puppet_facts_index` and change the definition to `index=puppet_facts_data`, click save |
46 | | -5. Select `puppet_metrics_index` and change the definition to `index=puppet_metrics_data`, click save |
47 | | -6. Select `puppet_detailed_index` and change the definition to `index=puppet_detailed_data`, click save |
48 | | -- Reload the main view of the Puppet Report Viewer app, and you should see data, or perform the following search: |
49 | | -``` |
50 | | -`puppet_all_index` sourcetype=puppet:* |
51 | | -``` |
| 1 | +# Puppet Report Viewer |
52 | 2 |
|
| 3 | +##### Table of Contents |
53 | 4 |
|
54 | | -Troubleshooting and verification |
55 | | ----------------- |
56 | | -If loading the Puppet Report Viewer app in the Splunk console does not appear to show any data after you have followed the setup guides for both this Splunk App/Add-on and the [splunk_hec module](https://github.com/puppetlabs/puppetlabs-splunk_hec), the first step is to check that data is being successfully sent to the Splunk server by following the troubleshooting and verification steps in the s[splunk_hec module](https://github.com/puppetlabs/puppetlabs-splunk_hec). |
| 5 | +1. [Description](#description) |
| 6 | +2. [Configuration](#configuration) |
| 7 | +3. [Advanced Configuration](#advanced-configuration) |
| 8 | +4. [Troubleshooting and Verification](#troubleshooting-and-verification) |
57 | 9 |
|
58 | | -If after verifying that Puppet is properly sending data to Splunk, the first issue may be that searches that populate the Splunk App aren't querying the right index. In Puppet Report Viewer versions released after 1.5.0, you can use the macro support described in Advanced Configuration options to set the index name. To find the index name, perfeorm a search by sourcetype across all indexes, and then click the disclosure `>` on the left side of the events and note the value used for `index` and provide that to the macro, do this for each sourcetype you are using (puppet:facts, puppet:summary, puppet:detailed are what are used in this app primarily as of this writing) if multiple indexes may be used. |
| 10 | +## Description |
59 | 11 |
|
60 | | -This is an example query: `index=* sourcetype=puppet:summary` |
| 12 | +This Splunk app provides views into the status of Puppet installations that are configured to send reports and metrics with the [`splunk_hec`](https://forge.puppet.com/puppetlabs/splunk_hec) and [`puppet_metrics_collector`](https://forge.puppet.com/puppetlabs/puppet_metrics_collector) Puppet modules. |
61 | 13 |
|
62 | | -If `puppet:detailed` sourcetype's are not showing up in search, that means that the Detailed Report Generator Alert is not configured or running properly. The app includes an alert that is disabled, but will trigger the alert as needed for any summary report that is submitted that isn't a normal, no change, puppet report. If this alert is enabled, and the setup screen has values in it, one can view the logs with the following Splunk search: |
| 14 | +## Configuration |
63 | 15 |
|
64 | | -`index=_internal sourcetype=splunkd component=sendmodalert (action="puppet_run_task_investigate" OR action="puppet_run_task" OR action="puppet_run_task_act" OR action="puppet_generate_detailed_report")` |
| 16 | +Once the application has been installed follow the steps below to configure the [Puppet Report Viewer](https://splunkbase.splunk.com/app/4413/): |
65 | 17 |
|
66 | | -There is also a view into the alert actions logs themselves (which are published to the HEC) under the Actions menu in the Splunk app that shows these searches as well. |
| 18 | +Create an Splunk HEC token for the app: |
67 | 19 |
|
68 | | -If there are no error messages, then verify that the HEC token supplied works and can be used to submit an example report properly, runnging this command from a unix or command line will submit a dummy puppet:detailed event (such as a Splunk server): |
| 20 | + * Select Puppet Report Viewer from the App dropdown in the Splunk console. |
| 21 | + * Navigate to `Settings` > `Data Input`. |
| 22 | + * Add a new `HTTP Event Collector` token with a name of your choice. |
| 23 | + * Ensure `indexer acknowledgement` is **not** enabled. |
| 24 | + * Click Next and set the source type to **Automatic**. |
| 25 | + * Add the `main` index |
| 26 | + * Set the **Default Index** to `main`. |
| 27 | + * Click **Review** and then **Submit**. |
| 28 | + * When complete the HEC token should look something like this: |
69 | 29 |
|
70 | | -``` |
71 | | -[cbarker@splunk-dev ~]$ curl -k -H "Authorization: Splunk yourHECtoken" https://localhost:8088/services/collector/event -d '{"sourcetype": "puppet:detailed", "event": "exampletest"}' |
72 | | -``` |
73 | | -The endpoint will respond with either a success or an error message, follow steps in the [HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) to resolve issues with the HEC. |
| 30 | +  |
74 | 31 |
|
75 | | -If there are other error messages in the logs related to authentication tokens from the Puppet side, this command will attempt to do a search of the PuppetDB system: |
| 32 | +After configuring both `splunk_hec` and the `puppet_metrics_collector` the Overview tab will start showing data from Puppet reports, while the Metrics tab will start displaying graphs related to a number of useful Puppet metrics. |
76 | 33 |
|
77 | | -``` |
78 | | -curl -k 'https://<your.puppetdb.server>:8081/pdb/query/v4/nodes' -H "X-Authentication: <token contents>" |
79 | | -``` |
| 34 | + |
80 | 35 |
|
81 | | -Responses will be in json, so piping the output through python may be helpful: |
| 36 | + |
82 | 37 |
|
| 38 | +## Advanced Configuration |
| 39 | + |
| 40 | +All dashboard views support using custom indexes for storing event data. The searches assume each sourcetype can be stored in it's own index. There is one top level macro (`puppet_index`) which defaults to `""`. |
| 41 | + |
| 42 | +As such, you can specify different HEC tokens for Summary Reports, Facts, and Metrics in `splunk_hec`. If you configure multiple HEC tokens to utilize different indexes, change the `index` value in the dashboards to reflect this. |
| 43 | + |
| 44 | +##### Example Configuration: |
| 45 | + |
| 46 | +Create five indexes named `puppet_data`, `puppet_summary_data`, `puppet_facts_data`, `puppet_metrics_data`, and `puppet_detailed_data`. |
| 47 | + |
| 48 | +Create five HEC tokens: |
| 49 | + |
| 50 | + * `puppet_data` with sourcetype of `automatic` and the index `puppet_data`. |
| 51 | + * `puppet_summary` with sourcetype of `puppet:summary` and the index `puppet_summary_data`. |
| 52 | + * `puppet_facts` with sourcetype of `puppet:facts` and the index of `puppet_facts_data`. |
| 53 | + * `puppet_metrics` with sourcetype of `puppet:metrics` and the index of `puppet_metrics_data`. |
| 54 | + * `puppet_detailed` with sourcetype of `puppet:detailed` and the index of `puppet_detailed_data`. |
| 55 | + |
| 56 | +Configure the `splunk_hec` module with the following token parameters: |
| 57 | + |
| 58 | + * `splunk_hec::token` with the `puppet_data` token value. |
| 59 | + * `splunk_hec::token_summary` with the `puppet_summary` token value. |
| 60 | + * `splunk_hec::token_facts` with the `puppet_facts` token value. |
| 61 | + * `splunk_hec::token_metrics` with the `puppet_metrics` token value. |
| 62 | + |
| 63 | +Update the search macros to use the new values: |
| 64 | + |
| 65 | +* Open **Advanced Search** under `Settings` > `Knowledge`. |
| 66 | +* Select `Search Macros`. |
| 67 | +* Select `puppet_index` and change the definition to `index=puppet_data`, click save. |
| 68 | +* Select `puppet_summary_index` and change the definition to `index=puppet_summary_data`, click save. |
| 69 | +* Select `puppet_facts_index` and change the definition to `index=puppet_facts_data`, click save. |
| 70 | +* Select `puppet_metrics_index` and change the definition to `index=puppet_metrics_data`, click save. |
| 71 | +* Select `puppet_detailed_index` and change the definition to `index=puppet_detailed_data`, click save. |
| 72 | + |
| 73 | +Upon reloading the **Overview** tab in the Puppet Report Viewer app, and you should begin seeing data. Alternatively, perform the following search: |
| 74 | + |
| 75 | +``` |
| 76 | +`puppet_all_index` sourcetype=puppet:* |
83 | 77 | ``` |
84 | | -curl -k 'https://<your.puppetdb.server>:8081/pdb/query/v4/nodes' -H "X-Authentication: <token contents>" | python -m json.tool |
85 | | -``` |
86 | 78 |
|
87 | | -Diagnose issues with PE auth tokens following [PE access documentation](https://puppet.com/docs/pe/latest/managing_access.html). |
| 79 | +## Troubleshooting and Verification |
| 80 | + |
| 81 | +If the Puppet Report Viewer does not appear to show any data after you have followed the configuration steps for both this app and the [splunk_hec](https://github.com/puppetlabs/puppetlabs-splunk_hec) module; first check that data is being successfully sent to the Splunk server by following the [troubleshooting and verification](https://github.com/puppetlabs/puppetlabs-splunk_hec/blob/main/docs/troubleshooting_and_verification.md) steps in the `splunk_hec` documentation. |
88 | 82 |
|
89 | | -Alert actions: |
| 83 | +If events in the `puppet:detailed` source type are not showing up in search, it means that the "Generate a Detailed Report" Alert is not configured properly with the [Puppet Alert Actions](https://splunkbase.splunk.com/app/4928/) add-on. If this Alert is enabled, and the aforementioned add-on is configured, you can view the logs with the following Splunk search: |
90 | 84 |
|
91 | | -To check the logs of the individual alerts being run you can search the internal log index in splunk: `index=_internal sourcetype=splunkd component=sendmodalert` however not all error messages will show depending on the debug level of your installation. |
| 85 | +``` |
| 86 | +index=_internal sourcetype=splunkd component=sendmodalert (action="puppet_run_task_investigate" OR action="puppet_run_task" OR action="puppet_run_task_act" OR action="puppet_generate_detailed_report") |
| 87 | +``` |
92 | 88 |
|
93 | | -Individual alert actions log to the the Splunk folder on the system running the actions also, so one can see more verbose logs in `/opt/splunk/var/log/splunk` |
| 89 | +There is also a view into the Alert Actions logs under the Actions tab which will show these searches as well. |
94 | 90 |
|
| 91 | +If there are no error messages, verify that the configured HEC works can be used to submit a report manually: |
| 92 | + |
| 93 | +``` |
| 94 | +$ curl -k -H "Authorization: Splunk <yourHECtoken>" https://localhost:8088/services/collector/event -d '{"sourcetype": "puppet:detailed", "event": "exampletest"}' |
| 95 | +``` |
| 96 | +The endpoint will respond with either a success or an error message. Follow steps in the [HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) documentation to resolve issues with the HEC endpoint. |
95 | 97 |
|
96 | | -More information |
97 | | ----------------- |
| 98 | +If there are other error messages in the logs related to PE RBAC tokens from the Puppet side, run the following command to query the PuppetDB API: |
98 | 99 |
|
99 | | -This addon will be updated frequently with more dashboards and views to data as feedback is gathered. Contact Puppet via the developer link and watch the [Puppet Community Office Hours calendar](https://puppet.com/community/office-hours) for future Splunk related events in our community [Slack](https://slack.puppet.com). |
100 | | - |
| 100 | +``` |
| 101 | +curl -k 'https://<your.puppetdb.server>:8081/pdb/query/v4/nodes' -H "X-Authentication: <token contents>" |
| 102 | +``` |
0 commit comments