Starting this blog

For the last two years I've worked on Splunk every day: onboarding data, fixing parsing, building dashboards and alerts, and more and more, shrinking what goes into the indexes without losing anything people search for.

Most of what I learn ends up in private notes. From now on, it goes here.

What I'll write about

• Cutting ingest with Ingest Processor and Edge Processor, and checking that nothing was lost
• Onboarding awkward data sources such as EDR, XDR, firewall and AWS logs
• Building Splunk add-ons and custom search commands in Python
• Observability with Splunk and Datadog
• What I learn while studying for the Splunk Power User and Admin certifications

A search to start with

When someone asks where the license is going, this is a good first look. It adds up the last seven days of indexed data by sourcetype, using the license usage log:

index=_internal source=*license_usage.log* type=Usage earliest=-7d@d
| stats sum(b) AS bytes BY st
| eval GB = round(bytes / 1024 / 1024 / 1024, 2)
| sort - GB
| rename st AS sourcetype
| fields sourcetype GB

Run it on the license manager, or on a search head that can search the license manager's _internal index. The sourcetypes at the top of the list are usually where an ingest pipeline will save the most.