When fast updates bite back

You already know that I am fast to apply updates and move to new products. I have almost all our WSUS updates set to auto approve and I load them on our servers and workstations the same day. I have a lot of faith and confidence in the patching process. But every once in a while, they bite back.

I can recall a few years ago that Microsoft got a lot of flack for blue screening computes with an update. I was keeping up to date with the situation from various news feeds. I unapproved the update while I investigated it more. We were not seeing any blue screens but other admins were. I saw all kinds of email flying around warning people and reporting issues.

As real details started to poor in, it turns out that the only computers that were blue screening were the ones with root kit infections. I immediately pushed that patch out to the rest of my computers. If my computers were infected, I wanted to know about it. In the end we had a clean bill of health but not everyone was so lucky.

I had the Bing desktop search bar get deployed once. Turns out I had feature packs on auto approve. I quickly fixed that and recalled it.

Last year Microsoft released a patch that would not trust 512bit certs anymore. I was following the progress of this issue for a while. The Flame malware was using a 512bit cert of Microsoft's that was weak enough to break. MS revoked that cert and later released this patch to break all 512 bit certs. I took a quick peek at our central IT's cert server. While I did see a few of those 512 bit certs, I saw many more 1024 and 2048 bit ones. I figured we had nothing to worry about.

Turns out that our email was using one of the weaker certs. So every one of my users was getting an error message that Outlook did not trust our email server. I got on the phone with central IT and pushed them to get an updated cert rolled out.  They recommended that the rest of the org not install that patch. it turned out that they needed to update a root cert first and that is kind of a delicate process when you don't do it very often. That was not something they were going to put a rush job on. Luckily Microsoft had a KB that talked about this issue and offered a command that would trust 512bit certs again.

I was able to Powershell that command out to everyone and life returned to normal. I was able to revert that setting once the certs were taken care of.

The one update that almost bit us the hardest was the Powershell 3.0 and remote management update that was released around December 2012. We started to run into some strange issues with remote Powershell and SCCM config man. And before we knew it, we realized that we could not remotely Powershell anything. SCCM was also down and out. I started to deep dive into the internals of WinRM to fix this. Listeners were broken and Powershell was refusing to re-register settings it needed for remote management.

Something reminded me that Powershell 3.0 was out and I found it on our workstations. We started finding reports of compatibility issues with SCCM 2012 and Powershell 3.0. Config Manager was attempting to repair WMI but would corrupt it instead. We ended up pulling that patch using WSUS and everything returned to normal in a few days. The SCCM server took a little more work to correct.

Not having Powershell when you need it can be very scary. That is my go to tool to recover from most issues. So handy to for a WSUS check in or gpupdate or ipconfig /flushdns to resolve some issue.

I think patching fast works well in our environment because we have a good team that is flexible and quick to respond to these types of issues. We still get caught off guard from time to time, but we handle it well. 

Lightning Fast Updates

How fast do you deploy updates? If Microsoft released a run of the mill update today, how soon would you see it on your production systems?

I like to patch my systems quickly. Over time, I have gotten quicker and quicker at rolling them out. WSUS was a great addition to Windows Server. Not only do I auto approve the important updates, I also auto approve just about everything else.  I found myself blindly approving them twice a year anyway.  That tends to create a monster patch. The problem with monster patches is that people notice them. The login takes longer so you tell them its the patches. Then if something like a hard drive goes out, then people blame the patches.

I still hold of on service packs, new versions of IE, and feature packs. All for good reason. I have only been caught off guard a few times because of it. So I have everything on auto approve.

One issue that I did have for a while was patches showing up later in the week than I expected. I would be ready for patch Tuesday expecting that things would be patched Wednesday morning and they were not. It felt like most of our patches hit on Thursday instead. So I started to look into it.

Our machines patch at 3:00 am if they are powered on. I tried to get WSUS to update just before 3:00am so things would be ready to go when the computers went to update. Sounded good in theory but that is not how the client updates. I found out that the computer will check in with WSUS once every 24 hours by default. If WSUS was not pulling updates until 3:00 am, then everything was really updating 24 hours behind.

So know I knew why my updates felt a whole day behind. I increased how many times WSUS would pull updates from Microsoft to 3 and let it run for a long time. My WSUS server was checking for updates at 11:00 am, 7:00pm, and 3:00am central. This way I was catching any other updates that showed up at odd times. I would get a few more machines and servers updated a day ahead, but the bulk of them was still 24 hours behind.

There is a very subtle detail here that I overlooked for the longest time. I had no idea what time Microsoft actually released at on Tuesdays. I ran with this schedule for a very long time. Then one day I was really reading an important IE patch that everyone was rushing to load and I saw the expected release time. It was at 10:00 pst. Seeing this time reminded me to check my update schedule.

Sure enough, I had it in my mind that they were released in the evening. I could see my logs showing updates getting pulled at 7:00pm every Tuesday patch day. With 10:00pst being 12:00 central, it clicked with me where my issue was.  I moved that early sync to 1:00pm and everything started updating right on schedule. All my servers and workstations were updating as expected with a very clear schedule.

I also started scheduling wake on lan to power up our workstations and combined it with a check for updates event. So now all my computers are getting updated as fast as reasonably possible and I know exactly when to expect issues.

AppLocker Audit Mode Three Months Later

I enabled AppLocker in audit mode about 3 months ago for all of our workstations. I spent about 2 weeks checking the logs and adding rules. I put it on the back burner to take care of some other things and almost forgot about it.  I ran those scripts I posted previously to check up on my workstations and things look fairly clean. Here are a few things that stand out to me.

There are a handful of things that run out of the user's profile and ProgramData that I need to be aware of.  I see a Citrix and WebEx client pop up on a few machines. Spotify also jumps out in the list. I didn't realize how many of our users used that. I also see a few Java updates being ran from the temp internet files folder. Nothing too crazy here that would have impacted much. I expect it would have been a hand full of panic calls from people that could not get some web conferences to work.

I did find a custom app that we wrote sitting on some desktops that would have broke. That would be been a big deal. I think I will just sign those apps and place them in the Program Files folder. I can use these logs to track down these users. This app is just an exe so there is no installer or registry thumbprints to look for.

The last group of findings were just a hand full of special machines that had something installed to a folder on the root of the C: drive. I could guess exactly where these machines were based on the names of those folders. I will handle these case by case. I am tempted to just give them local exceptions instead of baking something into the main policy.

Now that we are aware of these things, we can do things right going forward. Primarily loading everything into the program files would be the most help. I plan on letting this go for another several months and see what else I pick up.

Review AppLocker Logs with PowerShell Remoting

I ran AppLocker in audit mode for a few days on a small number of computers.  So all that activity is collecting in the "Microsoft-Windows-AppLocker/EXE and DLL" audit log.  It creates an event every time an application starts indicating if it was allowed, blocked, or would have been blocked.  That last event type is 8003 and that’s the one I care about.

The Powershell command to view this log entry is this:

get-winevent -logname "Microsoft-Windows-AppLocker/EXE and DLL" |

Where-Object{$_.id -eq 8003}  |

ft message

This will tell me every application that would have failed.  I can either make a new rule or ignore it knowing that it would be blocked in the future.  I can combine this with powershell remoting to check the event log on every computer I manage.

Get-QADComputer | %{Invoke-Command $_.Name –AsJob –ScriptBlock{

    $ErrorActionPreference = "SilentlyContinue"

    get-winevent -logname "Microsoft-Windows-AppLocker/EXE and DLL" |

     ?{$_.id -eq 8003} |

     Format-Table message

}}

 I use the QuestAD tools to get every computer in the domain and request the log event 8003 from the correct event log.  The other stuff just cleans up the output.  Give it 60 seconds to finish or timeout (for computers that are not powered up). Then run these commands for the results.

Get-Job | ?{$_.State -eq "Failed" -or $_.HasMoreData -eq $false} | Remove-Job

Get-Job | Receive-Job -Keep

 This will filter out results we don’t care about and then output all the logs on all the other systems. If you have pages of data, you can process them one computer at a time. This walks the results from the top down.

(Get-Job | ?{ $_.HasMoreData -eq $true})[0] | Receive-Job

 When it outputs the results, it will reset the HasMoreData flag from that Job.  So if you see some output and you want to know what job it was from, run Get-Job.  In the middle of the list, you will see the the HasMoreData flip from false to true.  The bottom one with a false value is the last computer you pulled output from.  This can be very handy when setting up rules.

If you have the admin share open to administrators, you can open explorer to \\computername\c$ and find files on it.  You can also use that remote admin share in the wizard to add new rules.

I saw Google Chrome show up on a computer in a user’s profile on a remote computer.  I was able to point the AppLocker rule wizard to \\computername\c$\users\john\appdata\.... and it added the needed rules.  I was able to add 4-5 needed applications.  I also saw some spyware on a few computers that I was able to clean up.

Now that we added some new rules, I wanted to clear the logs so they are cleaner next time.  Here is the command to do that.

Wevtutil.exe cl "Microsoft-Windows-AppLocker/EXE and DLL"

 I plan on repeating this ritual every few days to identify new rules. Eventually I will no longer be adding rules and can look at enforcing them without much risk.

Getting Started with AppLocker

I am only running this in audit mode and I am already finding benefits of using it.  AppLocker allows you to white list applications.  If you were to use this on workstations that did not grant administrator access, you could probably stop all malware without any other protection.  It turns out to be a lot easier than I thought. 

The idea of white listing every application felt like a daunting task.  There are a set of rules you can use to make this easier. Running the default rules in audit mode can give you a good idea of how much work it will take. If you use a consistent image for every workstation deployment and install everything in Program Files, then this gets very easy.

First we needs to enable the Application Identity Service. I enabled it in the same policy that I plan on configuring the rules in.

This should start on the next reboot. The next step is to configure auditing mode.

Now we need to create some rules.  Right click on Executable Rules and create default rules. This will create 3 important rules for you to prevent you from locking users of the computers.  The first is the Administrator rule allowing admins the ability to run anything.  The other two cover the Windows and Program Files folder.  Any file in those locations are allowed to run.

If your users are not local administrator on the workstations, then the only things that can be in those folders were programs installed as an administrator. This is a very important point that highlights why this works so well. The only rules you need to add are ones for non-standard programs that don’t run from Program Files. Hopefully this is a short list. 

There are three types of rules you will deal with.  Path rules, publisher rules, and checksum rules. The built in wizard does most of the work for this.  Just point it at your installed application and it will do the rest.  You have the option to make adjustments by hand if needed.

Now apply this policy to computers in Active Directory. Give your computer plenty of time to get a reboot and a few days of activity.

Powershell: Compare system configuration vs baseline

Powershell makes it very easy to take custom baselines and compare configurations to that baseline.  One of the easiest examples is checking for changes to services.  Powershell has simple cmdlets for listing services, saving the results to a file, and comparing for differences.  Here is a quick snip of code that hi-lights what we are doing.

$baseline = get-service
stop-service spooler -force
$current = get-service
Compare-Object $baseline $current -Property Name Status

This will capture a baseline, stop your print spooler, then capture a current list.  Then we compare a few properties.  It will hilight that in one list the service is running and in the other it is not.  If a new service was added or removed, then it would also be indicated.  We can use this simple concept to build a configuration change tracking system.

I want to expand this a little bit into a script that I can run every week to show me the changes on my systems. Sounds easy enough, so lets see what we come up with.

function Compare-Baseline($folder, $id, $command, $properties){

 $Root = "$folder"
 $report = "$root\$id-report.txt"

 #Prep folders
 if((Test-Path($Root)) -eq $false){
     md $Root
 }

 #Prep Baseline
 if((Test-Path("$root\$id-base.xml")) -eq $false  ){
     & $command | Select-Object -Property $properties | Export-Clixml -Path "$root\$id-base.xml"
     "New $id Baseline created" | Out-File $report -Append
 }

 & $command | Select-Object -Property $properties | Export-Clixml -Path "$root\$id-current.xml"

 $base = Import-Clixml $root\$id-base.xml
 $current = Import-Clixml $root\$id-current.xml
 $compare =  Compare-Object $base $current -Property $properties -SyncWindow 100 

 if($compare){
  $compare | Out-File $report -Append
 }
 Remove-Item $root\$id-base.xml
 Move-Item $root\$id-current.xml $root\$id-base.xml

 type $report
} 

Compare-Baseline "c:\scratch" "Service" {Get-WMIObject win32_service} ("Name", "Startmode", "state", "pathname")

Now I can run this any time I want to see when the services on this box change.  I decided to use the WMI win32_service because it gives me a few more details, the its the same idea.  I wrote this in a very general way so it would be possible run it on many machines.

I have several ideas for this going forward.  I could easily schedule this and have the results emailed to me.  I may also collect those baselines in a central location.  Taking this a step further, I can have one task that checks AD for servers.  Then runs this once on each server.  This would allow it to discover new servers and provide me a single report.