AUTOMATION 6 - Unit Check, Part 2: random distribution

We're looking at unit checking a Java dice rolling class - see last time for more information.


Previously by necessity we occasionally had to show some technical details, and raw Java code - I'm hoping those of you less technical are still with me.

Today will have less of that - I'll specify parts of code, so those of you experimenting with Eclipse yourselves can keep up and experiment, but it will be less important.

Distribution of numbers

Okay - I've got a test which assures me all the numbers 1-6 come up.  But that doesn't address my major concern, which is I'm worried that the number 1 will come up significantly less than numbers 2, 3, 4, 5 or 6.

How would I manually test this?  Well as before, I'd keep pressing "roll again" and keep a table of the results.  I'd expect them to look evenly spread, 1/6 each.  Ish.

Again - great test, but there's a clause there like above "I'd keep doing until I'm satisfied / dissatisfied".

Automation needs a hard and fast rule.  Welcome to the world of computers - things have to either be a 1 or a 0, a pass or a fail.  And "until I'm satisfied" is not hard or fast.

Now ... there's a something which is often quoted about random numbers.  If you sample over a large enough sample set, then you should see on your dice rolls a perfect split of 1/6 for each number using a 6 sided dice.

When I've manually tested this before (believe me I did statistics at University and I did this one night with my best friend David Farish), and notice that the results do smooth out a bit.  However I'd usually get bored at about 200 dice rolls.

However in my automated unit check, I have the power to sample over 6 million dice rolls.  Over such a huge number, everything should even off to a cool million each.  So I'm going to rename and reuse that code I had yesterday.

@Test public void randomNumberSpread() { System.out.println("Confirming even distribution"); int totalRolls = 6000000; ...

  //Confirm each number rolled 1/6 of the time for(counter = 0; counter < numSides; counter++) { assertTrue("Dice rolls for " + (counter+1) + " not perfect 1/6", (countArray[counter] == (totalRolls/6) ) ); }
}

Let's see how it works shall we?

Oh - that failed.  I thought over such a large number, it'd all even out.  When I ran this, I genuinely thought it might be within 1 or 2 of the target number.

Maybe I need to roll more.  Let's set our total rolls for 600 million, and see what happens?

It takes about 21 seconds to run, but ...

Maybe if I try again?

Well, obviously I've not run enough dice yet, let's set it to 60 billion!!!  And that dear readers is where I break my software completely because

Basically I've pushed the number of rolls so high, that Java can't handle unless I change a lot of the data types I'm using.

I could ... but I'm reminded of the third iron law,

And I'm starting to go down the path of making this check way too complex.  Just adding more dice rolls doesn't seem to help.

Here's the frustrating thing - look at the results from these three runs

Run 1

Run 2

Run 3

In Run 1, the dice rolls for numbers 1, 4 and 6 fall just under the 100 million "perfect" number.  In Run 2, the dice rolls for numbers 1 and 6 are just above 100 million.  Whilst on Run 3, the dice rolls for the number 4 is above 100 million.

Intuitively, using manual judgement, that says to me that I've seen numbers fall above and below the average, but only a little bit.  So that looks okay to me.

I've tried this test several times - sometimes it take 2 goes, sometimes 3, sometimes 4.  Fundamentally though the challenge is "how do I create a hard and fast rule for this to test?".

When I posed this problem on Twitter, Iain Mc pointed out to me

I tried to get a good calculated way to determine an expect deviation for this which led me to Yahoo answers and this ...

d^2 = [1/(N-1)]∑(xi - m)^2 = [1/(N-1)]*{∑[N*vi*P(vi)]^2 - 2*N*m*∑v*P(vi) +N*m^2}

d = √ [1/(N-1)]*{∑[N*vi*P(vi)]^2 - 2*N*m*∑vi*P(vi) +N*m^2}

At which point, I remembered that I've not done "hard maths" for a while, so maybe Iain had a point.

What I've decided to do is to modify the test.  Obviously running 600 million rolls doesn't superbly define the problem for me, and at 21 seconds, takes a bit too long.



What I've decided to do is to test a threshold of plus or minus 1%, which should be good enough.  And I'll do it over 600,000 dice rolls, which will be much faster (couple of seconds including compilation), but still nicely divisible by 6.

@Test public void randomNumberSpread() { System.out.println("Confirming even distribution"); int totalRolls = 600000; int minVal, maxVal;
...

//Work out min/max threshold as +/- 1%
minVal = (int) (0.99 * totalRolls / 6);
maxVal = (int) (1.01 * totalRolls / 6);

   //Confirm each number rolled 1/6 of the time for(counter = 0; counter < numSides; counter++) { assertTrue("Dice rolls for " + (counter+1) + " below threshold of " + minVal, (countArray[counter] > minVal ) ); assertTrue("Dice rolls for " + (counter+1) + " above threshold of " + maxVal, (countArray[counter] < maxVal ) ); }
}

And that seems to run nicely ... and so far repeatedly ...

Okay - that's a lot of maths and thinking.  We still have a little more to play with before we sum up, for now, we'll wrap up.

Recap

So we managed to create an important automated check - but it was a little harder than initially expected, because as the first iron law said, we needed to summarise what we wanted to check into a hard pass/fail criteria.  And that turned out to be harder than we intuitively first assumed.

Extension material

Have a think about what other tests you think might be worth running.  If you're using Eclipse, have a go with it.

But also try and think about problems with this for you as a tester.