rrdtool data resolution

One concept I had to adjust to was deciding on the resolution of the data I wanted to collect. By resolution I mean how detailed the data is. Let us use the example of CPU usage. If I intend to look at the last 1 hour vs the last 30 days, I would see the data at different details. My 1 hour window may show me every single data point (@5 sec). Every peak and dip is exactly what it was. Each pixel is 5 sec and I have 720 data points (60 min, 12 points a min).

Now when we look at the 30 days we will not be able to include that much detail on the screen at once. My 30 day window will show data points @60 minute ticks. If the CPU slowly goes up and down, that's not an issue. But that is not often the case. It is very spiky, up and down very quickly at times. If you only take a reading every 60 min, its any ones guess if you read a spike or a dip.

In this case I would record the min,max, and average values. When you chart that, the max will be the spikes to show you how hard it gets pushed. The min will show you if it ever gets to idle.

When you create a rrdtool database, you define all those options. You can define multiple counters to the same thing at different resolutions. You define a database and indicate how often the value will be recorded. Then you can define different resolutions of that data. You will still just record the CPU every 5 sec and the rrdtool will keep track of those data windows and resolutions.

We can define a database that gets a value every 5 sec for our CPU dataset. We can define a detailed resolution of 1 hour and a second one of 30 days. The first RRA records a value every tick for 720 ticks. each tick is 5 sec, so that's 1 hour. The RRA would be "RRA:AVERAGE:0.5:1:720". The second RRA averages the values of 720 values (1 hour) over 30 days (720 ticks). That RRA is "RRA:AVERAGE:0.5:720:720".

Let us pick some new numbers. What if we want 30 sec average for 2 days. 30 sec is 6 values of data. 2 days is 5760 intervals of 30 sec. So the RRA is "RRA:AVERAGE:0.5:6:5760".

So our final rrdtool database is this:

rrdtool create temp.rrd -s 5
DS:cpu:GAUGE:30:0:100
RRA:AVERAGE:0.5:1:720
RRA:AVERAGE:0.5:720:720
RRA:AVERAGE:0.5:6:5760

What I want you to do is read over this and then return to the examples on the rrdtool's website. Once this information clicks, come back here and read it again. I know the RRA code is over your head if this is your intro, but when you return it will be a very solid example.

Intro to rrdtool

I recently discovered a simple tool that has lots of power behind it. rrdtool is a round robin database that stores time dependant values and easily graphs them. It is a database where you insert values at consistent intervals and the query results are in the form of a graph. It is a round robin database because it only saves a set number of values and overwrites the oldest one every time.

Now that I pointed out what it is, let us talk about what we can do with that. The first thing that jumps out (and what it was designed for) is performance monitoring. You can set up a task to save the CPU, network, disk, and ram activity to this file every 5 sec and then generate charts to display it. Any thing you can get a counter on. Computer temperature, event log errors, ping times, terminal server connections, and anything else you can think of. Things like the daily temperature, number of visitors to your office, spam messages, or even your daily bank balance.

This is great for performance monitoring because the database automatically discards old data. To put it another way, it only keeps the data for as long as you think its important. Once you define how much you want to store and at what resolution, the database is then set in size. It never grows or shrinks.

RemoteApp Disconnected Because of a protocol error in Windows 7 RC1

The first thing I did after installing Windows 7 RC1 was to toss on my remote apps off of a test server. I have a 2008 Terminal Server that publishes a few common apps for seamless integration. I received a few dissconnects to the remote apps on both windows 7 RC1 boxes that I set up. Here is that message:

RemoteApp Disconnected: Because of a protocol error, this session will be disconnected. Please try connecting to the remote computer again.

It is too soon for me to pin this on the release canidate because I have only spot tested my terminal server set up. But a quick search of the message did not give me many results. As I get more details, I will update this post.

VB.Net Auto grow Textbox as text changes

I have several textboxes on a form that I would like them to grow as the user types text into them. Sometimes they will only put in a few words, other times it can be several lines of text. I expected this to be a very simple process but had a hard time finding a solution.


Several people did talk about using the TextBox.lines.count to figure out how tall to make the box. The problem with that is you must turn word wrap off for that to work. Sometimes you can do that. In my case, I needed the lines to wrap.


I did find references to Graphics.MeasurString(String,Font) as SizeF to figure out how long a string would be with a given font. I was about to overlook the effect different fonts and sizes would have had but it looks like this will solve both.


In the end, I took each line in the txt box. I calculated how many times each line would wrap and add that to my line count. Then used the Textbox.Font.Height to figure out how high each line would be. I also added 10 pixels to account for any padding at the top and bottom of the text box.


Here is the code I ended up with:

Dim numberOfLines As Integer = 0
Dim e As Graphics = Graphics.FromImage(New Bitmap(300, 300))
Dim StringSize As New SizeF
For Each item As String In tBox.Lines
StringSize = e.MeasureString(item, tBox.Font)
numberOfLines += Math.Floor(StringSize.Width / tBox.Width) + 1
Next
tBox.Height = numberOfLines * tBox.Font.Height + 10

I added a call to that to the TextChanged event handler for each text box. Don't forget that you have to adjust the other items on your form if you make the text box grow. It will overlap other elements if left unchecked.

Creating the Bad Anatomy raffle for Warcraft

Recently Blizzard changed some rules to Warcraft that now allows people to run casinos in game. They are crude and the operators fill the city spam. You don't know who you can trust but people still play.

An idea I had worked on in the past was to create a lottery for players in wow. We had a lot of it planned but ended not doing because the rules were unclear if it was allowed or not. Now that the rules have changed and we are ready to start it up. So I present to you Bad Anatomy's Cho'gall weekly raffle.

Our initial idea was for a lottery where people could pick numbers and we use real world event to pick the numbers (like the multi-state lottery numbers). But our biggest issue with that is it could take a long time for a winner. In the end we decided on a raffle.

For every 10G someone donates, we will give them one raffle ticket. The ticket is given just so the purchaser knows what his numbers are and adds to the realism of it. A character was created just to handle the raffle. So every message in and out from him is kept separate from all our other mail.

There are 2 parts that make this work. First is the in-game mod I had to develop to receive gold, issue tickets and track tickets, and to pick the winner. It initially started out easier then I thought it should be and ended slightly more challenging then I thought it would be. I may go into those details in a later post. I can run my character up to the mail box and issue a command that will allow him to open every message, collect the gold, calculate how many tickets someone gets, and then send each of them a message with their numbers.

The second part is the supporting website Bad Anatomy's Cho'gall weekly raffle . This is the feature that makes it real. People can check the site to see when the next raffle starts or ends, see the winners, and all the rules to the game. We took a domain name we already had (Bad Anatomy) and re purposed it for the raffle. I tried to give it a simple and clean look. I tried to look at lottery sites for examples and they all felt overwhelming with all the stuff they tried to pack onto one page. In the end, I created the look I wanted.

This experiment is just getting started so I cannot tell you how well people are liking it or even how long I will run it. Head over to Bad Anatomy's Cho'gall weekly raffle to see how its going.

Tracking Down RTE: 35601 Element Not Found

We recently turned on some new features of a product we were using and would get this RTE on a few records within the application. The number of problem records were small, the total number of records worked with in a day made them a regular occurrence.

I went over to our test environment. It was up to current patch levels, but the data was a few months old. Almost all of my broken records in live were broken in test. That gave me hope that its an existing data issue and not something actively breaking records.

The nice thing about our test environment is that I am usually the only person using it. That makes it easy to toss the SQL Profiler on with out having to filter it to just my traffic. I started the profiler at the point just before clicking the button that generated the RTE and paused it just after the message came up. I closed the app and repeated this process a few times for both working and broken records.

Looking at the queries that each ran, the broken records all stopped at the same point after the same query was run. That tells me that the application is processing the queries as it runs them and the last one before it stopped is my suspect query. If both the broken and the working records had the same full list of queries I would have had to check them all. That would have indicated to me that it pre-loads most of the data before processing it. I was in luck.

The query was a large one but the only difference between all records was the record ID. So it is the same query running for both working and non-working records. Running the query for all of them gave me different sized record sets. Some had more a lot of records and some had very few. One thing did jump out at me as I looked at the raw data.

A few null values jumped out to me in a column that looked like it should have a value. With my experience with the data I knew it should have had a value. I saw one column that had data in it directly related to reference I expected instead of that null value. Because this looked like a complicated query, my first guess was a bad join.

That column was not the only one with nulls in it but every time it had a null the 3 columns to the right also had nulls. That's what I expect a bad join to look like. So I started reformatting the query with proper indents and white space. It did contain a few union calls. The first query in the union practically returned the same results. It looked the same except for a few less rows. My null values were still mixed into the good data.

I focused on the table columns that were null and the one column that contained directly related information. The joins between them looked correct. Exactly how I would have written them. After rewriting the query as simple 2 table join that used the same logic, I ended up with the same results. It was a join that I expected to be valid every time but in this case there are times where its not.

It turned out to be a flaw in the application logic where it allowed some changes to one set of tables that should have enforced changes in another. But they were enforcing them at the application level instead. Other parts of the system enforced it correctly, this one point in the application did not. It overlooked the fact that it needed to enforce it and I almost expect the developer to make the same assumption I made.

I know this is a little different from my usual problem/solution posts but this was mostly an internal product and I wanted to document the process I took to solve it.

Quiet/Loud Game

I posted this in another forum and got a lot of good feedback on it. Here is a more formal write up on a game I play with my kids at home. The Quiet Loud Game.

I have 2 young kids at home. One is almost 4 and the other is 17 months. The 4 year old would often make lots on noise like kids do and our attempts to quiet her down were not working very well. Our problem was she didn't truly understand the concept of quiet. She knew she was doing something wrong, but had no idea what. That did not work very well for either of us. I came up with a new game to play.

I turn the radio on to some music and turn the volume up a bit. We run around the house being loud. I yell the work loud or noisy several times and we all jump, and stomp, and scream, and yell, and slam doors, and bang on the floor. Then we start a loud count down from 3, 2, 1 and then I whisper the word quiet as I turn the volume on the radio way down.

Then we all whisper, and tip toe, and sneak, and shush, and softly open and close doors. Only making quiet sounds. Then we start a soft count down from 3, 2, 1 and then I yell the work loud as I turn the radio back up.

We repeat this process several times until someone starts to wear out (usually me) and we end it while we are all quiet. I tell them they did a great job and we all give each other high fives.

Not only does this help teach them the concept of loud and quiet, it also gives them an outlet for that noise. And we all had a lot of fun doing it. I made a special point to do all the stuff we consider loud when doing the loud part.

Left/Right Airplane Game

My little girl loves airplaines. We live close enought to an airport that she can spot them in the sky all the time. Once of her favorite games is to fly like an airplaine on my sholder. I holder her up with her legs out behind and her arms out to the side. She tells me where to go as we fly around the house.

Recently I changed it so she has to say Left or Right. As soon as she says a direction, we go that way. Even if she is pointing the other way. After doing this a few time, she was pointing the same direction she was saying. It worked out well and we had fun doing it.

W32.Downadup.C is in the wild. It looks like Conflicker is starting to evolve.

It recently received a new set of instructions that are designed more to protect it then to make it spread any more. It is continuing to attack antivirus software that is used to clean it up. Just as the security industry has gotten into its system of communicating with its self, the virus has gotten a new algorithm that makes it 200 times harder.

This cat and mouse game is about to get interesting. We have yet to see a payload from this virus. There is no question that the attack has changed at this point. Without using any new attack to build a larger infection base, it has to hold what it has already. You don’t fight that hard to keep a system infected unless you have a plan for it later.

https://forums2.symantec.com/t5/Malicious-Code/W32-Downadup-C-Digs-in-Deeper/ba-p/393245%3bjsessionid=1A353B585C96A581ECB9D3536C31ADCB

VB Dates and DBNull.Value SQL

One think I find that I have to look up often is how to work with database dates in vb.net code. I can never remember the best way to do it and end up looking at older code all the time to do it.

Dates are not usually the problem as much as empty dates. Moving an empty date from vb to SQL and back again. I don't expect my solution to be the best solutions. Personally I feel its too complicated so there had to be a better solution that I just have not found yet. With that said, here is how I handle null dates in vb.net.

I don't like working with null values when crossing between applications and code. Mostly because a blank value can serve the same purpose with less code. Dates are the one place I can't use a blank value and it must be null. I will show the use of both strings and dates so you can easily see how they differ.

Here is how I grab values from a DataRow object called row:

dim name as string = row("Name").ToString()
dim DOB as Nullable(Of DateTime) = row.Field(Of Nullable(Of DateTime))("DOB")

My use of ToString() is a quick way to convert DBNull strings into a empty string. I could store my date as a string and do the same thing, but there are times I want to keep it as a datetime object. I could use a plain DateTime and just check the value as I read it for DBNull and adjust to that. Or I can use a Nullable DateTime object to do the same thing for me.

When I am saving the values, I do check for values when I add them as Parameters. Here is how that code looks like:


Dim command As New SqlClient.SqlCommand()

command.Parameters.AddWithValue("@Name", Name)
If DOB Is Nothing Then
command.Parameters.AddWithValue("@DOB", DBNull.Value)
Else
command.Parameters.AddWithValue("@DOB", DOB)
End If

One other thing that you need to watch is using DBNull values in a where statement. By default NULL=NULL is false. You have to check for IS NULL instead. If you want more details on why that is look up ANSI_NULLS. I mention this because its very common to want to use a parameter in a WHERE statement. If you don't craft it correctly, you will not get the results you expect when the value is NULL.

In your mind you expect this to work:

SELECT * FROM people WHERE DOB = @DOB

You want to use a DBNULL to find all the people without a DOB but it will return 0 records. I bet you could mess with the ANSI_NULL option or change the query to this:

SELECT * FROM people WHERE (DOB = @DOB OR (@DOB IS NULL AND DOB IS NULL))

This will do the normal check on the param, but will also check to see if both are null. I know I am doing a lot of extra work someplace, but this does work for me. Turning off the ANSI_NULL option will make the first query work, but I have not looked into it enough to know if it has any negative effects.

New transaction cannot enlist in the specified transaction coordinator

I was tying to set up a special feature in the software I work with that involved linked servers. This was test setup so I configured some SQL servers I already had set up and ran into this error.

[OLE/DB provider returned message: New transaction cannot enlist in the specified transaction coordinator. ]
OLE DB error trace [OLE/DB Provider 'SQLOLEDB' ITransactionJoin::JoinTransaction returned 0x8004d00a].
Msg 7391, Level 16, State 1, Line 2
The operation could not be performed because the OLE DB provider 'SQLOLEDB' was unable to begin a distributed transaction.

The only message the application gave me was the first line but I uncovered a sql command that let me test it without the overhead of running my application. Here is that command:


BEGIN DISTRIBUTED TRANSACTION
select * from linkname.databasename.dbo.tablename
COMMIT TRAN

I was able to get the inner command to run on its own. As a distributed transaction, it would fail every time.

I eventualy found this KB from microsoft to get me in the right direction. http://support.microsoft.com/kb/839279

You may receive a 7391 error message in SQL Server 2000 when you run a distributed transaction against a linked server after you install Windows Server 2003 or Windows XP Service Pack 2

It turned out to be a settings issue with the Microsoft Distributed Transation Cordinator (MS DTC). I had no idea what that realy was. It turns out MS DTC should be enabled when SQL is installed but by default its locked down (or is with 2003/xp sp2).

I used dcomcnfg to get to the component services. Console Root -> Component Services -> Computers -> My computer -> Properties . On the Default properties enable distributed COM on the computer. Under MSDTC -> Security Configuration check the option for Network DTC Access.

I did make those changes on both servers, I don't know if that was needed or not. I did the linked server last and it worked as soon as I applied the changes. I know those were the exact changes I made on the 2nd server. The first one prabably has every thing checked or opened up trying to figure it out.

Other thing you may try if having issues it to make sure the Distributed Transaction Coordinator is running as network service and not local system. Reinstalling it as a service and rebooting were a few other tips I read while trouble shooting this problem.

Single Level Active Directory Domain

If you are considering giving your Active Directory name just a single level name, don't do it. Just stop and pick a new name. If your already stuck with a single level AD domain, I feel your pain. I see the warning now when I try to fix my problems after the fact.

So whats the big deal? Windows XP and Server 2003 do not update DNS records to a single level domain. They also have issues joining a domain cross subnet. Our biggest problems show up when dealing with more then one subnet.

The main problem is that DNS will be missing records. Domain controllers have lots of integration with DNS. So if one of those is not updating its DNS records, you have a huge mess. Domain controllers failing to replicate was our big issue. Every time we added a domain controller, we ended up rebuilding the DNS records by hand. Adding the new server keys where needed. Most of the time replication would work one way but not the other. Fixing DNS fixed replication.

Over time I have uncovered more documentation and most importantly this registry key:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters]"UpdateTopLevelDomainZones"=dword:00000001

I run this on every server just to make sure. Once you run this, you will have to reboot the server so the netlogon process can register the DNS.

You can run this on your workstations if you want them in your DNS too. we opted not to for most of ours.

I believe there is now a group policy setting that sets the same value. But I do not recall where it is. If I find it, I will update this post.

The second big issue we ran into was we could not join the domain from the other subnets. In XP and 2003, Microsoft changed them to use DNS more. The computer could see the domain and tell you that it exists. It would even point out that the SRV records in DNS are correct. But it would fail to join. At some point in the process it sees the single level domain name and used netbios instead of DNS. It would work fine if on the same subnet, but with out a wins server or lmhost file it would fail.

For the longest time, we used a lmhost file to point to the domain. We built it into the default ghost image and it worked great. Then came the network restructure that ended up changing our IP range. Evey one of those file we used in the last several years had to be updated. The symptom was a 10 min login on those computers for existing users on the computer.

Here is a sample of what we put in the lmhost file

10.0.0.1 servername1 #PRE #DOM:doaminname
10.0.0.2 servername2 #PRE #DOM:domainname
10.0.0.1 "domainname \0x1b" #PRE
10.0.0.2 "domainname \0x1b" #PRE

Inside the quotes on the bottom line must be exactly 20 characters or it won't work.After saving the changes to the LMHOSTS file you have to enable NetBIOS over TCP/IP and import theLMHOSTS file.

The correct way to do this would be to set up a wins server. One experiment that I tried was to put one of the domain controllers address as the wins server and it solved our import issue. The only thing we needed a wins server for was to join computers to the domain on a different subnet. By telling the computer that the domain controller is the wins server, it was able to find it and import just fine.

So our work around was to run a registry hack on every server so they update DNS and to put in a fake wins server address that points to our domain controller for workstation importing.

Ventrilo - Ranks and Mute/Queue Options

There are lots of reasons you may or may not want to give users their own accounts on your vent server. I use it to give me more control over a large group. Let me give you an example.



I play alot of World of Warcraft and our raid groups have 25 members. A few need to be able to talk all the time while others just talk all the time and don't stop. If you get alot of people telling everyone how to do something their special way, it just makes it more complicated for everyone else.



So I have a raid channel set up that mutes all guests to that channel. All guild members have an account so they can talk. New and random people to the group we keep silent. We want them to listen to our method. If they have a sugestion, they can send it silently to a raid leader in game to relay to the group. This option is in the channel options under "Disable Guest Account Transmitting". Checking this will keep all guests from broadcasting in that channel.



Another thing you can do is give each member a rank and mute low ranked members in a channel. I experimented with this but it does not work as smooth as I would like it to work. Im taling about the voice mode on the channel. The default option is normal that allows all ranks to talk. The other 2 options are Queued and Muted. The each act a little different but will keep ranks under a set level from broadcasting.



Queued: This one will mute everyone else while someone is talking. So only one person can broadcast at a time. It uses the Transmit Rank Level to only allow set ranks to broadcast at all. The person that gets the mic keeps it until he stops broadcasting. Once he stops, the next new broadcast gets it. The catch is if someone starts to broadcast just before they last person has released his mic, they will stay muted. I don't like this setting because someone could think they are eaying something important, but because they never got the green light it never broadcasted. I feel like I have to watch vent on a 2nd monitor to use this. The exception here is if you only have one person that will ever talk in this setting. You can adjust this setting on the fly.

Ventrilo - How to connect?

Ventrilo is a very popular voice chat program. It is used with many games to cordinate the actions of many players at once. The first step to using Vent is to get connected.

First step is to create a user name. This is the name that everyone will see when you connect. You can create a phonetic for it so it will announce when you join or leave the channel to everyone there (that have not turned that annoying feature off). Be respectfull with that because it follows you from server to server and its easy to forget you have one set up.

If its a new connection, you will need to add the server before you can connect to it. You will usualy be provided a server/port/password to enter. If they do not provide a password, you prabably do not need one.

The server can be a port or an IP address. It can look like guild.typefrag.com or 10.120.34.4. You may see the port number attached to the server name with a colin. If the port number is 12345 its possible it can look like this; guild.typefrag.com:12345 or 10.120.34.4:12345. If you are not given a port number, look closer at the sever name.

Once you add the server you can click connect. If everything is correct, you should find yourself in the root channel. From here you can double click on a channel to join it.

Ventrilo Guide on its way

I am doing alot of work with Ventrilo and finding the information available on the features I wan't to use either don't exist or are hard to find. It's my plan to document some of these things to make it easier for others.

For now check out this guide: http://www.trap17.com/forums/Ventrilo-Gamers-Friends-t61530.html