3D dice results are not random

[Weissrolf]

After comparing FGU's "physical" D8 dice results with FGU's own "non-physical" /die command I concluded that FGU's physical dice engine seems to be bugged and does not create random numbers. FGU's non-physical /die pseudo-random number generator (PRNG) has its own flaws, but it is mostly functional compared to the current 3D implementation (judging by D8 rolls).

No, non-physical dice and physical dice use completely different systems.

As a short-term hotfix I suggest to disable the 3D dice engine all together, as quickly as possible. As a long-term improvement I suggest to implement a better PRNG (like PCG) and give users the option to disable the 3D dice physics engine if it cannot be improved to match the /die engine.

If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random.

Chi square distribution for 375000 samples is 13393.71, and randomly would exceed this value less than 0.01 percent of the times.

Failures on short-ranged tests, especially DC6 parameterizations, are typical of small chaotic PRNGs with insufficient mixing / avalanche.

length= 8 kilobytes (2^13 bytes), time= 0.2 seconds
  Test Name                         Raw       Processed     Evaluation
  BCFN(2+0,13-9,T)                  R= +29.3  p =  7.5e-8   very suspicious
  DC6-9x1Bytes-1                    R= +30.1  p =  2.9e-13    FAIL
  ...and 23 test result(s) without anomalies

BCFN failures where the first parameter listed is low are typically similar in meaning to a failure on a DC6 parameterization.

length= 8 kilobytes (2^13 bytes), time= 1.3 seconds
  Test Name                         Raw       Processed     Evaluation
  BCFN(0+0,13-9,T)                  R= +65.6  p =  2.7e-16    FAIL !
  DC6-9x1Bytes-1                    R= +30.1  p =  2.9e-13    FAIL
  DC6-6x2Bytes-1                    R= +27.4  p =  1.9e-12    FAIL
  DC6-5x4Bytes-1                    R= +34.7  p =  3.4e-15    FAIL !

Failures on FPF parameterizations can have similar meaning.

length= 32 kilobytes (2^15 bytes), time= 0.3 seconds
  Test Name                         Raw       Processed     Evaluation
  BCFN(0+0,13-9,T)                  R=+244.9  p =  3.8e-56    FAIL !!!!
  BCFN(0+1,13-9,T)                  R=+271.2  p =  4.6e-62    FAIL !!!!
  BCFN(0+2,13-9,T)                  R=+143.0  p =  2.6e-34    FAIL !!!
  DC6-9x1Bytes-1                    R= +75.6  p =  1.5e-37    FAIL !!!
  DC6-6x2Bytes-1                    R= +99.0  p =  5.4e-41    FAIL !!!
  DC6-5x4Bytes-1                    R=+109.0  p =  1.1e-57    FAIL !!!!
  FPF-14+6/32:(0,14-8)              R= +14.1  p =  3.4e-10  very suspicious
  FPF-14+6/32:(1,14-9)              R= +11.2  p =  2.6e-7   unusual
  FPF-14+6/32:all                   R= +12.2  p =  1.2e-7   very suspicious
  FPF-14+6/16:(0,14-8)              R= +22.4  p =  4.0e-16    FAIL
  FPF-14+6/16:(1,14-8)              R= +22.3  p =  4.1e-16    FAIL
  FPF-14+6/16:(2,14-9)              R= +14.0  p =  3.9e-9   suspicious
  FPF-14+6/16:all                   R= +28.0  p =  2.5e-18    FAIL !
  FPF-14+6/16:cross                 R=  +7.3  p =  1.6e-5   mildly suspicious
  FPF-14+6/4:(0,14-6)               R= +47.5  p =  2.2e-36    FAIL !!!
  FPF-14+6/4:(1,14-7)               R= +29.8  p =  1.6e-23    FAIL !!
  FPF-14+6/4:(2,14-8)               R= +38.1  p =  1.9e-27    FAIL !!
  FPF-14+6/4:(3,14-8)               R= +12.1  p =  9.6e-9   suspicious
  FPF-14+6/4:all                    R= +63.7  p =  3.9e-50    FAIL !!!!
  FPF-14+6/4:cross                  R= +28.6  p =  1.1e-20    FAIL !!

The graph shows the distribution of the ordered p-values obtained from each sub-test; there are two points (one point for KS and one point for AD) for each valid sub-test in the "Table" and in the "Pearson" pages. When a good generator is tested, the points shouldn't be too far from the black ideal line.

https://www.fourmilab.ch/random/
https://pracrand.sourceforge.net/
https://cristianopi.altervista.org/RaBiGeTe_MT/

https://en.wikipedia.org/wiki/Randomness_tests

[Weissrolf]

Let me clarify this up-front: Please stay on topic of the PRNG engine. Don't be defensive, but open to discussion. I have no interest in seeing FGU fail at such a fundamental mechanic, but want to see it fixed and improved. Thanks in advance!

[Kelrugem]

Ah, interesting

But I think some explanation of the data is needed, for example, what is shown in the last graphic? The axes are not labelled etc. A bit difficult to follow without explanation what is calculated

[LordEntrails]

I also don't know how to read that.

From your 200 d8 examples, I would expect something like 25 of each result. Does your sample data show a range of 24-26 results for each value, or is range more like 5-50?

[Weissrolf]

I quoted the explanations, but here are the links to the corresponding web-sites. For the graph I added the labeling of the axes in the quote now, but all we really need to know here is that the lines are not even broadly close to each other and that FGU's /die tests considerably better. It's just another visualization of how very badly FGU's physical engine fails.

A simpler and better visualization is in the first image that demonstrates how FGU's physical D8 ebb and flow within only 200 rolls. That's why I put it first and the other one last.

https://www.fourmilab.ch/random/
https://pracrand.sourceforge.net/
https://cristianopi.altervista.org/RaBiGeTe_MT/

https://en.wikipedia.org/wiki/Randomness_tests

[Weissrolf]

I also don't know how to read that.

If it says "Fail" in too many tests then it's not random. Especially Practrand usually needs a huuuuge amount of data (like terabytes) to falsify randomness of good RNGs, but FGU's D8 data is so problematic that is manages with just a few kilobytes. FGU's /die data also fails, but later and at a different test (pointing to different problems).

From your 200 d8 examples, I would expect something like 25 of each result. Does your sample data show a range of 24-26 results for each value, or is range more like 5-50?

Sorry, but I really don't understand the question?! What range do you mean?

What I did was to create binary data out of the dice rolls where each byte consist of numbers 0-255. Since FGU's physical dice cannot roll D256(-1) I rolled D8(-1) instead, stripped the five 00000 high bits and formed one long bitstream of 375000 bytes (=1 mio rolls of D8). Each D8's binary representation is between 000 to 111, 2.67 D8 make one byte between 00000000 to 11111111. For comparison I did the very same with random.org rolls of 0-7.

The first image is a bitmap where each binary 0 is black and each binary 1 is white. I found out that there is a period of only 600 pixels/bits where you see the ebb and flow of dark and light areas repeat in the bitmap image. 600 bits = 75 bytes x 2.67 = 200 rolls of D8. The image does not tell us which results were rolled, but it tells us that there are periodic ups and downs of more 1 bits vs. more 0 bits in the stream of consecutive rolls.

Again, not everyone has to understand what this all means, all you need to see is that both PCG and random.org look like random noise (the kind of static noise you may remember from old TVs), but FGU's D8 rolls show a clear periodic pattern.

[Kelrugem]

I quoted the explanations, but here are the links to the corresponding web-sites. For the graph I added the labeling of the axes in the quote, but all we really need to know here is that FGU's /die tests considerably better overall and that both PCG32 PRNG and random.org TRNG test even better than that. It's just another visualization of how very badly FGU's physical engine fails.

A simpler and better visualization is in the first image that demonstrates how FGU's physical D8 ebb and flow within only 200 rolls. That's why I put it first and the other one last.

https://www.fourmilab.ch/random/
https://pracrand.sourceforge.net/
https://cristianopi.altervista.org/RaBiGeTe_MT/

https://en.wikipedia.org/wiki/Randomness_tests

I mean it would be nice when just every technical term is shortly introduced with its definition, to avoid that one needs to go through the links to understand that So, basically just an introduction of what you did and what the values represent, and how to read that, not just listing the result like Fail or Success That would certainly improve readability for people who are not experienced with all these technical terms, basically it would be good when the reader can interpret the result on one's own (now one simply needs to trust the Fail message). I understand though when that's annoying to do, because you did it in your free time etc.

[Weissrolf]

It's rather impossible to do. We are not scientists here who even understand what this all means beyond fail and success. There just is no way to give a "short introduction". These are well established randomness tests and all we can do is trust their results, just like scientists around the world trust them and test them. Everyone wanting to go deeper needs to read through manuals and web-sites of scientists/coders who create and test random number generators (and even their disputes in papers and forums), and possibly study mathematics.

The first bitmap image is the most down-to-earth representation of FGU's PRNG failure I can give you. It is pure visualization of the numbers FGU rolled as black/white dots and should display random noise. Instead it displays visibly repeated patterns. That must not happen!

I did far more work on this than I should have done, but this represents a real number-crunching examination of FGU's physical dice results, free from any "I rolled too many crits today" bias. Not psychology, but mathematics. Does it go over my own head? Sure, but that's why Practrand tries to categorize it in readable fails and suspicions for dummies like us.

[Weissrolf]

And again, what really matters is the comparison to better (FGU /die) and proper (PCG, random.org) working RNGs. Even if we don't understand a thing about all the results we can easily see that one fails hard where others succeed.

I specifically did not include specific FGU /die results in the discussion yet, because physical 3D dice fails so much worse that we need to concentrate on them first.

[smelton]

I did far more work on this than I should have done,

Indeed. You compiled all this data comparing apples to oranges and then came here to conclude that not only are apples not oranges, there is something wrong with our apple because is it not an orange and we should get rid of it ...