Saturday, April 05, 2014

Fixing print drivers missing dependent files

I was having an issue with print drivers that felt corrupt. My quick way to fix a broken machine was to reset the spooler and delete the driver quickly in print management before it was locked by the spooler. Some times it took a few tries but it worked.

I continued to investigate the issue and it lead me to a set of registry keys that defined the print drivers along with the dependent files. My broken computers had a null key for the dependent files. I was able to write a script to identify the broken drivers.

#Quick check for bad printer driver
$keys = Get-ChildItem 'HKLM:\system\CurrentControlSet\Control\Print\Environments\Windows NT x86\Drivers\Version-3\'
    $keys | ?{$_.Name -notmatch "Microsoft enhanced Point and Print compatibility driver|Amyuni Document|Microsoft Office|OneNote|Foxit Reader"} |
        ?{ !((Get-ItemProperty -Path ("Registry::" + $"Dependent Files")}

I figured out that I could delete the driver key from this location and the spooler would download the correct driver from the print spooler after it was restarted. Delete key and restart the service. I ended up with this script:

#Quick fix bad printer driver
$keys = Get-ChildItem 'HKLM:\system\CurrentControlSet\Control\Print\Environments\Windows NT x86\Drivers\Version-3\'
$keys | ?{$_.Name -notmatch "Microsoft enhanced Point and Print compatibility driver|Amyuni Document|Microsoft Office|OneNote|Foxit Reader"} |
    ?{ !((Get-ItemProperty -Path ("Registry::" + $"Dependent Files")} |
    %{ ($_ | Remove-Item); Restart-Service spooler -force}

The nice thing about an identify and repair script like this is that I can run this remotely on all the computers that we have. It turns out that 10% of our computers had this issue.

I investigated several of these before I fixed them. I found issues from printing silently failing, to printing artifacts, to printers ignoring settings (like print dual sided), to advanced settings that were just missing. I think this issue is accounting for a lot more service calls that I initially gave it credit for.

I now have a script that I can run building wide to fix this problem. While that is good, I still don't have a fix that prevents the problem.

There are several threads talking about this issue and I think Microsoft has a patch in the works.

Friday, March 28, 2014

My printer drivers are corrupt

I hate dealing with printers. I know a lot of system administrators feel the same way. We recently started to see an increase in printer related calls. Some of them turned out to be a real pain to fix. Almost as if the driver was corrupted. It had to be a system issue because our users are not administrators. We either push the printer out or they add it from the print server. Not much for them to mess up.

We had one printer related issue that took the team too long to fix, so I found myself in the middle of it. Not just fixing the problem but also calming down the customer. Although we mostly had her fixed, someone else on the same printer had just broken. The issue they had was the advanced options to their Xerox printer were not available. They could not change paper trays or print dual-sided. The types of issues that IT often does not see as a big deal but it is very important to the customer that it works.

I walked over to a third office where everything was working. I checked the driver and it was the same as the others. I then printed a test page from all three of them. Not only was the driver the same, it was even the same version number. But something very important jumped out to me. The list of dependent files only printed out from the computer that was working correctly. So it looked like they were missing all the additional printer driver files. 

I delete the printer and reconnected again for good measure and the test print was the same. I pulled up the printer management tool and tried to delete the driver, but it said it was in use. Access denied. Disconnected the printer and rebooted for good measure. It was still a problem. I finally got it removed by restarting the spooler and deleting the driver as fast as I could. It took me 5 attempts, but I got it. Connected to the printer and everything was working correctly. Did the same on the first broken user and it fixed them too.

Now I have a sure fire fix to the problem. It looks like IT witchcraft though. Stop spooler, start spooler, and delete driver. Repeat as needed. Sometimes it works the first time but most of the time it takes a few attempts. The user is usually sitting there with me.

Over the next few days, I fix a few more this same way. I want to dig into it deeper but printer drivers only break when someone need to print. The one day I had 3 break all within 10 minutes of each other near the end of the day. It was getting clear that we need to understand the issue better.

That very next day, one of the earlier users broke again. This time I made a copy of the spooler folder before the fix and after the fix. It turned out all the missing files were still there. Checksums were the same too. There goes the idea that the drivers are corrupt. For some reason Windows has the driver files but does not know it.

Edit: I was able to write a script to help with this issue:

Monday, March 03, 2014

Over the last 10 years

Interim Director of Information Systems and Learning Resources 02/2013 to present
Developer/Systems Analyst II 07/2006 to 02/2013
Workstation specialist 2003 to 07/2006

Proposed and implemented a 4-5 year equipment refresh policy. Architected and implemented a virtual desktop solution to reduce expenses. Implemented Server 2012R2 deduplication and storage spaces to extend the life of existing storage equipment and to acquire new storage at reasonable prices. Repurposed existing hardware for our backup plan to include full workstation backups for the entire college. Backup and disaster recovery systems were revised for better reliability and security.

Serve on the Leadership Council (2013-Current), Clinic Implementation Committee (2005-Current), Joint Privacy and Information Security Workgroup (2013-Current), End-User Device Security Team (2010-Current). Recipient of the Chancellor Council Silver Award of Excellence in 2012 for my role in the electronic dental record implementation.

Before serving as the Director, I filled many roles within Information Systems. As a Systems Administrator I architected, deployed, and administered Active Directory, extensive Group Policy Objects, DNS, DHCP, SQL Server clusters and availability groups, Hyper-V virtualization clusters, terminal servers, clinic system applications, fileservers and related share structures. Acquired and deployed into production entry level enterprise SAN for SQL and Hyper-V.

Owned the mission critical applications and systems. Served as the only database administrator managing vendor and internal databases. Handled system upgrades from Server 2003, to 2008R2, to 2012/2012R2 across our infrastructure. Executed the SQL cluster upgrades and hardware refreshes to our clinic systems from SQL Server 2000, to 2005, to 2008, to 2012. Implemented SQL Reporting Services (SSRS) and laid the foundation for a future data warehouse implementation.

Performed the initial implementation of Active Directory as a disaster recovery solution in 2004. Managed the migration of our users, computers, group policies, servers, and custom applications to another university domain in 2011. Assisted with the analysis, implementation, and ongoing support for a replacement clinic system in 2005 and for the digital record modules of that system in 2012. Implemented secure prescription printing solution. Performed the server side management and clinical integration of a MiPacs digital radiography system.

As a Developer, I work mostly with PowerShell, SQL, .Net, and aspx. My implementation of SSRS for reporting was to complement our clinic system. Worked closely with the business manager to define and implement the financial reports they required. Significant effort was also put into implementing grading, insurance, and research reports for the various areas of the college.

Developed other applications that interacted with the clinic system to increase user productivity. Many of my add-ons were used by the vendor at other sites or the features were integrated directly into the product. Some of those enhancements include a calendar style schedule for the clinicians, inventory management for the central dispensary, chart management for the records clerks, and a streamlined patient check in dashboard used by reception.

Wrote the design documents and was the project manager for a biomonitoring application we outsourced to Dell. Developed a DRM video player to protect human cadaver videos. Wrote an electronic insurance integration component for an internal clinic application. Built CD/DVD splash screens. Wrote USB activity auditing processes and various other odds and ends as solutions to issues the college was facing.

As a Workstation Specialist, I migrated us from XP to Windows 7 starting in fall of 2009. I assisted in setting up the imaging server and helped craft our master images. I am a strong advocate for least privileges so I spearheaded the effort to remove administrator rights from our users. I was able to leverage my developer experience to resolve issues applications had with either Windows 7 or restricted access. I now direct my own team on our Windows 8 and Virtual Desktop deployments.

Monday, November 25, 2013

Dell touch pad freezes on my Latitude Z

I recently started fighting an issue where my mouse pointer will become unresponsive on the screen. The right and left click still work. But the touch pad just stops on me. The only time I have this happen is after I wake my computer up. Not sure if it is from sleep or hibernate.

It is kind of hard to troubleshoot mouse issues when you can't use the mouse to help you. I can reboot the computer, but I would rather not. I am bad about leaving a lot of windows open and I try not to reboot if I can help it.

The strange thing about it was the dell touch pad software in the system tray could see me touching the touch pad. It shows a mini touch pad and it would highlight where my fingers were. The first thing I decided to do was to close that app. I then ran into my first challenge. How do I get focus to the task bar and send the right click command to the app?

I decided it was just easier to fire up powershell and kill the process. That was easy enough to do, but it didn't solve anything.

My next idea was to remove the driver and scan for hardware changes.It was a bit of a challenge using the device manager with out the mouse. I removed every mouse, keyboard, and HID driver I could find. I thought this worked once, but I could not get any results the next few times I tried that.

I then tried something else. I decided to kill every process one at a time on my computer until it either started working or it crashed. I already had powershell open from before. I typed in this command:

Get-Process |  %{$; $_.close(); sleep 3}

It took every process, printed the name, closed it, and then waited for 3 seconds. I did it this way so I could tell what process was closed just before the mouse started working. Best of all, it that it worked. It killed about 8 or so processes before it got to csrss. My mouse sprang back to life and I stopped the script before it killed too many processes on my machine.

I still don't know why that process is a problem. The next time this happens, then this will be the first place I look.

Wednesday, November 06, 2013

RRDTool Server Dashboards

I started to use RRDTool to record and chart performance counters of my servers. Here is a dashboard for one system I deal with. There are about 6 servers involved. Some counters apply to different servers.

There is a lot going on in this chart. It shows terminal server sessions, iops, available ram, cpu, SQL connections, and sql batchs. Together, these are the key counters that really show the health of the system. I don't report network activity because it is not as important in this context.

If you look at this next one for a single file server, I look at different counters. The nice thing about using RRDTool to generate graphs is that you can really build something customized to your environment.

The last one I am going to post is of my Hyper-V cluster. The storage iScsi connections are at the top for each individual node. Across the middle is the aggregate activity of all hosts and guests to the rest of the network. The bottom shows iops on the SAN and available ram for the cluster.

This is still a work in progress but these charts are already providing a lot of value and insight into our servers.

Using RRDTool to graph pings to a host.

I started to use RRDTool again for system monitoring. One item I wanted to track and graph is ping latency to a host. First thing we need to do is create a rrd file to hold the data.

    .\rrdtool.exe create datafile.rrd -s 15  DS:data:GAUGE:900:0:U  RRA:AVERAGE:0.5:1:2020    RRA:MAX:0.5:1:2020  
This is a basic file that will hold about a week worth of pings at a resolution of 15 seconds. The next step is to start collecting data.

            Test-Connection -ComputerName Computername -Count 1 -BufferSize 1460 |
                ForEach-Object{ .\rrdtool.exe update datafile.rrd N:$($_.ResponseTime)}
             Sleep 14

I jumped right in with this one. Ping the host forever every 14 seconds and record it in our datafile. I know we defined the file as receiving samples every 15 seconds but it is ok if we record more often. RRDTool will aggregate the values for us. Our next step is to produce a graph.

.\rrdtool.exe graph graph.png --start -15 -u 20 -l 0 -w 400 -h 300 -t "Pings to host" DEF:"A0"="datafile.rrd":data:AVERAGE AREA:A0#FF000022 LINE2:A0#FF0000:"Ping to host"

The graph will be 400x300 over the last 15 minutes with the average ping in red. It may look something like the one below. I had to extend it out 4 hours to show a little variety in ping times.

RRDTool ping graph sample
I took this further and created a dashboard for all of my servers. I customized the charts so that they looked green if the host is up and red if the host goes offline.
The graph command for something like this gets a little more complicated, but it highlights what you can do with RRDTool. This graph shows 5 minutes of time and goes gray after 24 hours of downtime. Here is the code for those charts if you are interested. 

Get-Content .\PingList.txt | ForEach-Object {
.\rrdtool.exe graph .\Graphs\$ -s now-300s -e now-25s -u 5 -l 0 -w 200 -h 60 -t "$_" -c BACK#000000 -c FONT#00ff00 -c CANVAS#FF0000 -c GRID#000000 -c MGRID#000000 -c ARROW#000000 -c AXIS#00ff00 -c SHADEA#000000 -c SHADEB#000000 `
    DEF:"A0"=".\ServersIV\$":data:AVERAGE `
    DEF:"PS"=".\ServersIV\$":data:AVERAGE:end=now-24h:start=end-24h `
    SHIFT:PS:86400 `
    CDEF:A0T=A0`,1`,+ `
    CDEF:A0TU=A0T`,UN `
    CDEF:PST=PS`,1`,+`,UN `
    TICK:MM#FFCC00:.5 `
    TICK:PSMM#333333:1 `
    TICK:A0T#339933:1 `
    AREA:A0#00000088 `
    LINE2:A0#000000  ;

Friday, October 04, 2013

Quick Script: Add permissions to registry key with Powershell

Write-Output "Setting Registry Permissions"
$acl = Get-Acl "HKLM:\SOFTWARE\App"
$person = [System.Security.Principal.NTAccount]"DOMAIN\Domain Users"
$access = [System.Security.AccessControl.RegistryRights]"FullControl"
$inheritance = [System.Security.AccessControl.InheritanceFlags]"ContainerInherit"
$propagation = [System.Security.AccessControl.PropagationFlags]"None"
$type = [System.Security.AccessControl.AccessControlType]"Allow"
$rule = New-Object System.Security.AccessControl.RegistryAccessRule("$person","$access","$inheritance","$propagation","$type")


$acl |Set-Acl -Path "HKLM:\SOFTWARE\App"