This issue can easily slip by unnoticed. All of that status indicators still show green, so everything is good, right? Not so! Statistics and gauge widgets will simply show that last statistic is received, whether it is recent or not. However, if you try to graph statistical data, there won't be any there. Several of the graph widgets will simply say "No data to display" as well.
I have a custom query based alert that lets me know if the CPU stats haven't received updates in the last hour to help shed light on situations like this. When I discovered this issue, I noticed that the SolarWinds Orion agents would randomly stop transmitting statistics even though they showed as connected. It was one of those random and intermittent issues that's difficult to reproduce until this issue has already arisen. Because of this, I spent between 6 and 9 months months working with SolarWinds developers and support to track down the issue and get a fix in place for it.
Below is a fix which finally resolved 95% of these agent issues.
Run NotePad as an Administrator and open the following file "c:\Program Files (x86)\Common Files\SolarWinds\JobEngine.v2\SWJobEngineSvc2.exe.config". Search for jobEngineSettings_Full and add the option emptyRouterCleanupInterval="-00:00:00.0010000" before the closing bracket so that it looks like the section below.
<jobEngineSettings_Full maxWorkerProcess="10"
jobExecutionRetention="01:00:00"
jobExecutionEnabled="false"
thresholdStreamByteThreshold="131072"
maxWorkerProcessLogFileAge="5.00:00:00"
maxHungJobs="1"
jobHungTimeout="00:00:30"
maxJobThreadsPerWorkerProcess="100"
maxJobThreadsPerWorkerProcess64="1000"
workerMonitoringPeriod="00:00:01.0000000"
forceSeparateWorkersOnAgent="true"
jobErrorSuppressionInterval="01:00:00"
jobErrorSuppressionCleanupInterval="00:05:00"
emptyRouterCleanupInterval="-00:00:00.0010000"
/>
After you make these configuration changes, you will need to restart the SolarWinds JobEngine v2 service.
This fix has to be repeated on each of your polling engines.
