Monday, November 25, 2013

Dell touch pad freezes on my Latitude Z

I recently started fighting an issue where my mouse pointer will become unresponsive on the screen. The right and left click still work. But the touch pad just stops on me. The only time I have this happen is after I wake my computer up. Not sure if it is from sleep or hibernate.

It is kind of hard to troubleshoot mouse issues when you can't use the mouse to help you. I can reboot the computer, but I would rather not. I am bad about leaving a lot of windows open and I try not to reboot if I can help it.

The strange thing about it was the dell touch pad software in the system tray could see me touching the touch pad. It shows a mini touch pad and it would highlight where my fingers were. The first thing I decided to do was to close that app. I then ran into my first challenge. How do I get focus to the task bar and send the right click command to the app?

I decided it was just easier to fire up powershell and kill the process. That was easy enough to do, but it didn't solve anything.

My next idea was to remove the driver and scan for hardware changes.It was a bit of a challenge using the device manager with out the mouse. I removed every mouse, keyboard, and HID driver I could find. I thought this worked once, but I could not get any results the next few times I tried that.

I then tried something else. I decided to kill every process one at a time on my computer until it either started working or it crashed. I already had powershell open from before. I typed in this command:

Get-Process |  %{$; $_.close(); sleep 3}

It took every process, printed the name, closed it, and then waited for 3 seconds. I did it this way so I could tell what process was closed just before the mouse started working. Best of all, it that it worked. It killed about 8 or so processes before it got to csrss. My mouse sprang back to life and I stopped the script before it killed too many processes on my machine.

I still don't know why that process is a problem. The next time this happens, then this will be the first place I look.

Wednesday, November 06, 2013

RRDTool Server Dashboards

I started to use RRDTool to record and chart performance counters of my servers. Here is a dashboard for one system I deal with. There are about 6 servers involved. Some counters apply to different servers.

There is a lot going on in this chart. It shows terminal server sessions, iops, available ram, cpu, SQL connections, and sql batchs. Together, these are the key counters that really show the health of the system. I don't report network activity because it is not as important in this context.

If you look at this next one for a single file server, I look at different counters. The nice thing about using RRDTool to generate graphs is that you can really build something customized to your environment.

The last one I am going to post is of my Hyper-V cluster. The storage iScsi connections are at the top for each individual node. Across the middle is the aggregate activity of all hosts and guests to the rest of the network. The bottom shows iops on the SAN and available ram for the cluster.

This is still a work in progress but these charts are already providing a lot of value and insight into our servers.

Using RRDTool to graph pings to a host.

I started to use RRDTool again for system monitoring. One item I wanted to track and graph is ping latency to a host. First thing we need to do is create a rrd file to hold the data.

    .\rrdtool.exe create datafile.rrd -s 15  DS:data:GAUGE:900:0:U  RRA:AVERAGE:0.5:1:2020    RRA:MAX:0.5:1:2020  
This is a basic file that will hold about a week worth of pings at a resolution of 15 seconds. The next step is to start collecting data.

            Test-Connection -ComputerName Computername -Count 1 -BufferSize 1460 |
                ForEach-Object{ .\rrdtool.exe update datafile.rrd N:$($_.ResponseTime)}
             Sleep 14

I jumped right in with this one. Ping the host forever every 14 seconds and record it in our datafile. I know we defined the file as receiving samples every 15 seconds but it is ok if we record more often. RRDTool will aggregate the values for us. Our next step is to produce a graph.

.\rrdtool.exe graph graph.png --start -15 -u 20 -l 0 -w 400 -h 300 -t "Pings to host" DEF:"A0"="datafile.rrd":data:AVERAGE AREA:A0#FF000022 LINE2:A0#FF0000:"Ping to host"

The graph will be 400x300 over the last 15 minutes with the average ping in red. It may look something like the one below. I had to extend it out 4 hours to show a little variety in ping times.

RRDTool ping graph sample
I took this further and created a dashboard for all of my servers. I customized the charts so that they looked green if the host is up and red if the host goes offline.
The graph command for something like this gets a little more complicated, but it highlights what you can do with RRDTool. This graph shows 5 minutes of time and goes gray after 24 hours of downtime. Here is the code for those charts if you are interested. 

Get-Content .\PingList.txt | ForEach-Object {
.\rrdtool.exe graph .\Graphs\$ -s now-300s -e now-25s -u 5 -l 0 -w 200 -h 60 -t "$_" -c BACK#000000 -c FONT#00ff00 -c CANVAS#FF0000 -c GRID#000000 -c MGRID#000000 -c ARROW#000000 -c AXIS#00ff00 -c SHADEA#000000 -c SHADEB#000000 `
    DEF:"A0"=".\ServersIV\$":data:AVERAGE `
    DEF:"PS"=".\ServersIV\$":data:AVERAGE:end=now-24h:start=end-24h `
    SHIFT:PS:86400 `
    CDEF:A0T=A0`,1`,+ `
    CDEF:A0TU=A0T`,UN `
    CDEF:PST=PS`,1`,+`,UN `
    TICK:MM#FFCC00:.5 `
    TICK:PSMM#333333:1 `
    TICK:A0T#339933:1 `
    AREA:A0#00000088 `
    LINE2:A0#000000  ;