iScsi SendTarget issues with MD3620i and VMM

I have a small Hyper-V cluster with 3 Dell R610s and a MD3620i storage array using 10G iScsi. The event log on the MD3600 unit generates an informational event every 30 minutes.  This makes the log very hard to read and slow to load.

Here is the full event:

Event type: 180C

Description: iSCSI connection terminated unexpectedly

Event specific codes: 0/0/0

Event category: Internal

Component type: iSCSI Initiator

Component location: 

Logged by: RAID Controller Module in slot 0

As I was tracking this down, I started investigating the event log on my host servers.  I found several MSiSCSI 113 warnings every half hour:  

Log Name:      System

Source:        MSiSCSI

Event ID:      113

Task Category: None

Description:

iSCSI discovery via SendTargets failed with error code 0x00001068 to target portal *192.168.1.2 0003260 Root\ISCSIPRT\0000_0 .

The warning would repeat for every target portal IP address of the MD3600.

After a lot of digging on the internet, I discovered this forum post: SCVMM 2008 R2 - Host Refresh causes Event ID 113 MSiSCSI events on Hyper-V Cluster. VMM will run a refresh on the cluster every 30 min and that refresh generates those errors.  It looks like the iSCSI paths are checked and target discovery is ran.  The solution at the bottom of the thread by bellacotim resolved this issue for me.

The problem is that, even though not all iSCSI HBA instances can actually reach the target in question, the user had set up the Discovery Portal to issue iSCSI "Send Targets" along all possible iSCSI HBAs + the MSFT SW initator.  This is the default behavior if all one does is specify the specific initiator.

To properly configure discovery, do the following (assumes a fresh environment):

• Open the iSCSI Initiator GUI
• Select the Discovery Tab
• Click "Discovery Portal..." button to open the Discovery Target Portal dialog
• Enter the IP address (optionally TCP Port number) of the target's iSCSI portal
• Click "Advanced..." button to open the Advanced Settings dialog
• On the "Local Adapter:" pulldown, select a specific HBA instance you *know* can actually connect to the target.  Hint:  By inspecting the list of IPs for this HBA instance (see 7 below), one can gain this knowledge
• On the "Initiator IP:" pulldown, select the local address from which this HBA should connect from
• Click OK to close the Advanced Settings dialog
• Click OK to save your changes
• Repeat from (3) for all Initiator - Target combinations

I would perform these steps during a maintenance window. It gives a warning about disconnecting active sessions when removing existing discovery targets. 

The good news from what I can tell is that its only filling up event logs with clutter.  As far as I can tell, it is not causing any performance issues. It also looks like it shows up connecting to the other MD units when using iSCSI.  MD3000i MD3200i MD3220i MD3600i MD3620i

Powershell: Compare system configuration vs baseline

Powershell makes it very easy to take custom baselines and compare configurations to that baseline.  One of the easiest examples is checking for changes to services.  Powershell has simple cmdlets for listing services, saving the results to a file, and comparing for differences.  Here is a quick snip of code that hi-lights what we are doing.

$baseline = get-service
stop-service spooler -force
$current = get-service
Compare-Object $baseline $current -Property Name Status

This will capture a baseline, stop your print spooler, then capture a current list.  Then we compare a few properties.  It will hilight that in one list the service is running and in the other it is not.  If a new service was added or removed, then it would also be indicated.  We can use this simple concept to build a configuration change tracking system.

I want to expand this a little bit into a script that I can run every week to show me the changes on my systems. Sounds easy enough, so lets see what we come up with.

function Compare-Baseline($folder, $id, $command, $properties){

 $Root = "$folder"
 $report = "$root\$id-report.txt"

 #Prep folders
 if((Test-Path($Root)) -eq $false){
     md $Root
 }

 #Prep Baseline
 if((Test-Path("$root\$id-base.xml")) -eq $false  ){
     & $command | Select-Object -Property $properties | Export-Clixml -Path "$root\$id-base.xml"
     "New $id Baseline created" | Out-File $report -Append
 }

 & $command | Select-Object -Property $properties | Export-Clixml -Path "$root\$id-current.xml"

 $base = Import-Clixml $root\$id-base.xml
 $current = Import-Clixml $root\$id-current.xml
 $compare =  Compare-Object $base $current -Property $properties -SyncWindow 100 

 if($compare){
  $compare | Out-File $report -Append
 }
 Remove-Item $root\$id-base.xml
 Move-Item $root\$id-current.xml $root\$id-base.xml

 type $report
} 

Compare-Baseline "c:\scratch" "Service" {Get-WMIObject win32_service} ("Name", "Startmode", "state", "pathname")

Now I can run this any time I want to see when the services on this box change.  I decided to use the WMI win32_service because it gives me a few more details, the its the same idea.  I wrote this in a very general way so it would be possible run it on many machines.

I have several ideas for this going forward.  I could easily schedule this and have the results emailed to me.  I may also collect those baselines in a central location.  Taking this a step further, I can have one task that checks AD for servers.  Then runs this once on each server.  This would allow it to discover new servers and provide me a single report.

rrdtool data resolution

One concept I had to adjust to was deciding on the resolution of the data I wanted to collect. By resolution I mean how detailed the data is. Let us use the example of CPU usage. If I intend to look at the last 1 hour vs the last 30 days, I would see the data at different details. My 1 hour window may show me every single data point (@5 sec). Every peak and dip is exactly what it was. Each pixel is 5 sec and I have 720 data points (60 min, 12 points a min).

Now when we look at the 30 days we will not be able to include that much detail on the screen at once. My 30 day window will show data points @60 minute ticks. If the CPU slowly goes up and down, that's not an issue. But that is not often the case. It is very spiky, up and down very quickly at times. If you only take a reading every 60 min, its any ones guess if you read a spike or a dip.

In this case I would record the min,max, and average values. When you chart that, the max will be the spikes to show you how hard it gets pushed. The min will show you if it ever gets to idle.

When you create a rrdtool database, you define all those options. You can define multiple counters to the same thing at different resolutions. You define a database and indicate how often the value will be recorded. Then you can define different resolutions of that data. You will still just record the CPU every 5 sec and the rrdtool will keep track of those data windows and resolutions.

We can define a database that gets a value every 5 sec for our CPU dataset. We can define a detailed resolution of 1 hour and a second one of 30 days. The first RRA records a value every tick for 720 ticks. each tick is 5 sec, so that's 1 hour. The RRA would be "RRA:AVERAGE:0.5:1:720". The second RRA averages the values of 720 values (1 hour) over 30 days (720 ticks). That RRA is "RRA:AVERAGE:0.5:720:720".

Let us pick some new numbers. What if we want 30 sec average for 2 days. 30 sec is 6 values of data. 2 days is 5760 intervals of 30 sec. So the RRA is "RRA:AVERAGE:0.5:6:5760".

So our final rrdtool database is this:

rrdtool create temp.rrd -s 5
DS:cpu:GAUGE:30:0:100
RRA:AVERAGE:0.5:1:720
RRA:AVERAGE:0.5:720:720
RRA:AVERAGE:0.5:6:5760

What I want you to do is read over this and then return to the examples on the rrdtool's website. Once this information clicks, come back here and read it again. I know the RRA code is over your head if this is your intro, but when you return it will be a very solid example.

Intro to rrdtool

I recently discovered a simple tool that has lots of power behind it. rrdtool is a round robin database that stores time dependant values and easily graphs them. It is a database where you insert values at consistent intervals and the query results are in the form of a graph. It is a round robin database because it only saves a set number of values and overwrites the oldest one every time.

Now that I pointed out what it is, let us talk about what we can do with that. The first thing that jumps out (and what it was designed for) is performance monitoring. You can set up a task to save the CPU, network, disk, and ram activity to this file every 5 sec and then generate charts to display it. Any thing you can get a counter on. Computer temperature, event log errors, ping times, terminal server connections, and anything else you can think of. Things like the daily temperature, number of visitors to your office, spam messages, or even your daily bank balance.

This is great for performance monitoring because the database automatically discards old data. To put it another way, it only keeps the data for as long as you think its important. Once you define how much you want to store and at what resolution, the database is then set in size. It never grows or shrinks.

RemoteApp Disconnected Because of a protocol error in Windows 7 RC1

The first thing I did after installing Windows 7 RC1 was to toss on my remote apps off of a test server. I have a 2008 Terminal Server that publishes a few common apps for seamless integration. I received a few dissconnects to the remote apps on both windows 7 RC1 boxes that I set up. Here is that message:

RemoteApp Disconnected: Because of a protocol error, this session will be disconnected. Please try connecting to the remote computer again.

It is too soon for me to pin this on the release canidate because I have only spot tested my terminal server set up. But a quick search of the message did not give me many results. As I get more details, I will update this post.

VB.Net Auto grow Textbox as text changes

I have several textboxes on a form that I would like them to grow as the user types text into them. Sometimes they will only put in a few words, other times it can be several lines of text. I expected this to be a very simple process but had a hard time finding a solution.


Several people did talk about using the TextBox.lines.count to figure out how tall to make the box. The problem with that is you must turn word wrap off for that to work. Sometimes you can do that. In my case, I needed the lines to wrap.


I did find references to Graphics.MeasurString(String,Font) as SizeF to figure out how long a string would be with a given font. I was about to overlook the effect different fonts and sizes would have had but it looks like this will solve both.


In the end, I took each line in the txt box. I calculated how many times each line would wrap and add that to my line count. Then used the Textbox.Font.Height to figure out how high each line would be. I also added 10 pixels to account for any padding at the top and bottom of the text box.


Here is the code I ended up with:

Dim numberOfLines As Integer = 0
Dim e As Graphics = Graphics.FromImage(New Bitmap(300, 300))
Dim StringSize As New SizeF
For Each item As String In tBox.Lines
StringSize = e.MeasureString(item, tBox.Font)
numberOfLines += Math.Floor(StringSize.Width / tBox.Width) + 1
Next
tBox.Height = numberOfLines * tBox.Font.Height + 10

I added a call to that to the TextChanged event handler for each text box. Don't forget that you have to adjust the other items on your form if you make the text box grow. It will overlap other elements if left unchecked.

Creating the Bad Anatomy raffle for Warcraft

Recently Blizzard changed some rules to Warcraft that now allows people to run casinos in game. They are crude and the operators fill the city spam. You don't know who you can trust but people still play.

An idea I had worked on in the past was to create a lottery for players in wow. We had a lot of it planned but ended not doing because the rules were unclear if it was allowed or not. Now that the rules have changed and we are ready to start it up. So I present to you Bad Anatomy's Cho'gall weekly raffle.

Our initial idea was for a lottery where people could pick numbers and we use real world event to pick the numbers (like the multi-state lottery numbers). But our biggest issue with that is it could take a long time for a winner. In the end we decided on a raffle.

For every 10G someone donates, we will give them one raffle ticket. The ticket is given just so the purchaser knows what his numbers are and adds to the realism of it. A character was created just to handle the raffle. So every message in and out from him is kept separate from all our other mail.

There are 2 parts that make this work. First is the in-game mod I had to develop to receive gold, issue tickets and track tickets, and to pick the winner. It initially started out easier then I thought it should be and ended slightly more challenging then I thought it would be. I may go into those details in a later post. I can run my character up to the mail box and issue a command that will allow him to open every message, collect the gold, calculate how many tickets someone gets, and then send each of them a message with their numbers.

The second part is the supporting website Bad Anatomy's Cho'gall weekly raffle . This is the feature that makes it real. People can check the site to see when the next raffle starts or ends, see the winners, and all the rules to the game. We took a domain name we already had (Bad Anatomy) and re purposed it for the raffle. I tried to give it a simple and clean look. I tried to look at lottery sites for examples and they all felt overwhelming with all the stuff they tried to pack onto one page. In the end, I created the look I wanted.

This experiment is just getting started so I cannot tell you how well people are liking it or even how long I will run it. Head over to Bad Anatomy's Cho'gall weekly raffle to see how its going.