Week 1: Backgrounder
While perhaps not the flashiest or most riveting subject, log files hold a lot of information about the performance and health of any system and Tableau is no different. While some logs are just noise, most (if not all) issues can be diagnosed just by investigating the log file. More importantly, the analysis is augmented when one combines logs with many seemingly unrelated other elements of the system (memory, cpu, disk, perfmon counters, etc).
If you’re bored, read through this article: http://ajax.sys-con.com/node/3141925. In it, the author mentions another ‘famous/popular’ article by Jay Krep — both highlight the value in log data and log analytics. It’s still fascinating to me that, despite its enormous benefit, log data is so out of reach and, gasp, out of mind for most analysts and engineers.
For the Tableau Logs (yes, it’s a big “L” for a reason), I’m going to be doing a month long series on what are the most important log files to follow: Backgrounder, Apache, VizQL and Data Engine. Alas, not all of them are pretty; most are text (with the exception of the vizql logs) so we’ll need to get creative with our log analytics tools and brush up on some of our regular expression knowledge. Don’t be afraid, however, the log tool we leverage makes searching though these files as easy as writing SQL so it, literally, is as simple as writing: where(/Superstore/i).
Where’s the Log
From the Tableau Server admin guide:
The Tableau Server log directory is C:\ProgramData\Tableau\Tableau Server\data\tabsvc\logs if you installed Tableau Server on drive C, unless otherwise noted in the table below.
Backgrounder, then, is in the folder: backgrounder. You’ll want to follow the *.log files.
What does it do for me?
Again, according to the docs, ‘Logs related to subscriptions and scheduled activities like extract refreshes, “Run Now” tasks, and tabcmd tasks.’
But there’s so much more. Some things (but not all) we watch for:
- Extract failures
- File delete failures
- Backgrounder process start/stop
- Extract size as it’s stored in the repository
For example, if you’re using a log analytics tool, you might be able to run a query like this:
where(/[WARN|INFO] com.tableausoftware\.(?P<type>(\w+\S+))/ AND type=model.workgroup.service.subscriptions.SubscriptionRunner) calculate(count)
With the above query (which we run on Logentries), we can get information on all subscription activities and any potential errors related to it.
If you’re really feeling up to it, you can leverage regular expression field extraction and make some custom grouping out of the text file.
In this example, let’s say you wanted to do a quick check on extract refresh amounts just by using a REST API to query your logs. Here’s what you send with the API call:
where(/Running job of type :(?P<TsJob>\w+)/) groupby(TsJob) sort(desc)
Yes, I know there are other ways to do that but (and here’s where it all becomes pretty powerful), you can send these as alerts as they happen to users via email and/or Slack. Or you can make this as part of a performance summary (via a Tableau Dashboard) about your Tableau infrastructure. There really is no limit when dealing with the log data; rather, it’s just making sure you don’t go insane reading through the noise.
What should I be Alerted on?
As with all “alerting/text messaging/email (basically anything that forces you to switch context)”, there is a rule: too many wrong alerts, you miss the right alerts. We integrate with Slack and leverage Tableau for summary views of the data.
For the backgrounder logs, we alert on 2 log bits:
- Backgrounder process starting/stopping
- Subscriptions to certain workbooks (getting difference between total time and run time)
While most alerts in the backgrounder logs aren’t mission critical, it’s still nice to know how often extracts refresh, fail, total time for subscriptions, and the like. And, as we start putting together the large Tableau Log Infrastructure, you’ll see how these logs start to connect to give a bigger picture about performance and health.
Stay tuned for next week where we’ll discuss everything about the Apache logs (httpd).