3D dice results are not random [Archive] - Fantasy Grounds Message Boards

Weissrolf

November 29th, 2020, 00:46

After comparing FGU's "physical" D8 dice results with FGU's own "non-physical" /die command I concluded that FGU's physical dice engine seems to be bugged and does not create random numbers. FGU's non-physical /die pseudo-random number generator (PRNG) has its own flaws, but it is mostly functional compared to the current 3D implementation (judging by D8 rolls).

No, non-physical dice and physical dice use completely different systems.

As a short-term hotfix I suggest to disable the 3D dice engine all together, as quickly as possible. As a long-term improvement I suggest to implement a better PRNG (like PCG) and give users the option to disable the 3D dice physics engine if it cannot be improved to match the /die engine.

https://i.imgur.com/0Yyhs5P.png https://i.imgur.com/BqXKT06.gif

If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random.

Chi square distribution for 375000 samples is 13393.71, and randomly would exceed this value less than 0.01 percent of the times.

Failures on short-ranged tests, especially DC6 parameterizations, are typical of small chaotic PRNGs with insufficient mixing / avalanche.

length= 8 kilobytes (2^13 bytes), time= 0.2 seconds
Test Name Raw Processed Evaluation
BCFN(2+0,13-9,T) R= +29.3 p = 7.5e-8 very suspicious
DC6-9x1Bytes-1 R= +30.1 p = 2.9e-13 FAIL
...and 23 test result(s) without anomalies

BCFN failures where the first parameter listed is low are typically similar in meaning to a failure on a DC6 parameterization.

length= 8 kilobytes (2^13 bytes), time= 1.3 seconds
Test Name Raw Processed Evaluation
BCFN(0+0,13-9,T) R= +65.6 p = 2.7e-16 FAIL !
DC6-9x1Bytes-1 R= +30.1 p = 2.9e-13 FAIL
DC6-6x2Bytes-1 R= +27.4 p = 1.9e-12 FAIL
DC6-5x4Bytes-1 R= +34.7 p = 3.4e-15 FAIL !

Failures on FPF parameterizations can have similar meaning.

length= 32 kilobytes (2^15 bytes), time= 0.3 seconds
Test Name Raw Processed Evaluation
BCFN(0+0,13-9,T) R=+244.9 p = 3.8e-56 FAIL !!!!
BCFN(0+1,13-9,T) R=+271.2 p = 4.6e-62 FAIL !!!!
BCFN(0+2,13-9,T) R=+143.0 p = 2.6e-34 FAIL !!!
DC6-9x1Bytes-1 R= +75.6 p = 1.5e-37 FAIL !!!
DC6-6x2Bytes-1 R= +99.0 p = 5.4e-41 FAIL !!!
DC6-5x4Bytes-1 R=+109.0 p = 1.1e-57 FAIL !!!!
FPF-14+6/32:(0,14-8) R= +14.1 p = 3.4e-10 very suspicious
FPF-14+6/32:(1,14-9) R= +11.2 p = 2.6e-7 unusual
FPF-14+6/32:all R= +12.2 p = 1.2e-7 very suspicious
FPF-14+6/16:(0,14-8) R= +22.4 p = 4.0e-16 FAIL
FPF-14+6/16:(1,14-8) R= +22.3 p = 4.1e-16 FAIL
FPF-14+6/16:(2,14-9) R= +14.0 p = 3.9e-9 suspicious
FPF-14+6/16:all R= +28.0 p = 2.5e-18 FAIL !
FPF-14+6/16:cross R= +7.3 p = 1.6e-5 mildly suspicious
FPF-14+6/4:(0,14-6) R= +47.5 p = 2.2e-36 FAIL !!!
FPF-14+6/4:(1,14-7) R= +29.8 p = 1.6e-23 FAIL !!
FPF-14+6/4:(2,14-8) R= +38.1 p = 1.9e-27 FAIL !!
FPF-14+6/4:(3,14-8) R= +12.1 p = 9.6e-9 suspicious
FPF-14+6/4:all R= +63.7 p = 3.9e-50 FAIL !!!!
FPF-14+6/4:cross R= +28.6 p = 1.1e-20 FAIL !!

The graph shows the distribution of the ordered p-values obtained from each sub-test; there are two points (one point for KS and one point for AD) for each valid sub-test in the "Table" and in the "Pearson" pages. When a good generator is tested, the points shouldn't be too far from the black ideal line.
https://i.imgur.com/cy4JO2b.png

https://www.fourmilab.ch/random/
https://pracrand.sourceforge.net/
https://cristianopi.altervista.org/RaBiGeTe_MT/

https://en.wikipedia.org/wiki/Randomness_tests

Weissrolf

November 29th, 2020, 00:56

Let me clarify this up-front: Please stay on topic of the PRNG engine. Don't be defensive, but open to discussion. I have no interest in seeing FGU fail at such a fundamental mechanic, but want to see it fixed and improved. Thanks in advance!

Kelrugem

November 29th, 2020, 01:18

Ah, interesting :)

But I think some explanation of the data is needed, for example, what is shown in the last graphic? The axes are not labelled etc. :) A bit difficult to follow without explanation what is calculated :)

LordEntrails

November 29th, 2020, 01:23

I also don't know how to read that.

From your 200 d8 examples, I would expect something like 25 of each result. Does your sample data show a range of 24-26 results for each value, or is range more like 5-50?

Weissrolf

November 29th, 2020, 01:26

I quoted the explanations, but here are the links to the corresponding web-sites. For the graph I added the labeling of the axes in the quote now, but all we really need to know here is that the lines are not even broadly close to each other and that FGU's /die tests considerably better. It's just another visualization of how very badly FGU's physical engine fails.

A simpler and better visualization is in the first image that demonstrates how FGU's physical D8 ebb and flow within only 200 rolls. That's why I put it first and the other one last.

https://www.fourmilab.ch/random/
https://pracrand.sourceforge.net/
https://cristianopi.altervista.org/RaBiGeTe_MT/

https://en.wikipedia.org/wiki/Randomness_tests

Weissrolf

November 29th, 2020, 01:34

I also don't know how to read that.
If it says "Fail" in too many tests then it's not random. Especially Practrand usually needs a huuuuge amount of data (like terabytes) to falsify randomness of good RNGs, but FGU's D8 data is so problematic that is manages with just a few kilobytes. FGU's /die data also fails, but later and at a different test (pointing to different problems).

From your 200 d8 examples, I would expect something like 25 of each result. Does your sample data show a range of 24-26 results for each value, or is range more like 5-50?
Sorry, but I really don't understand the question?! What range do you mean?

What I did was to create binary data out of the dice rolls where each byte consist of numbers 0-255. Since FGU's physical dice cannot roll D256(-1) I rolled D8(-1) instead, stripped the five 00000 high bits and formed one long bitstream of 375000 bytes (=1 mio rolls of D8). Each D8's binary representation is between 000 to 111, 2.67 D8 make one byte between 00000000 to 11111111. For comparison I did the very same with random.org rolls of 0-7.

The first image is a bitmap where each binary 0 is black and each binary 1 is white. I found out that there is a period of only 600 pixels/bits where you see the ebb and flow of dark and light areas repeat in the bitmap image. 600 bits = 75 bytes x 2.67 = 200 rolls of D8. The image does not tell us which results were rolled, but it tells us that there are periodic ups and downs of more 1 bits vs. more 0 bits in the stream of consecutive rolls.

Again, not everyone has to understand what this all means, all you need to see is that both PCG and random.org look like random noise (the kind of static noise you may remember from old TVs), but FGU's D8 rolls show a clear periodic pattern.

Kelrugem

November 29th, 2020, 01:48

I quoted the explanations, but here are the links to the corresponding web-sites. For the graph I added the labeling of the axes in the quote, but all we really need to know here is that FGU's /die tests considerably better overall and that both PCG32 PRNG and random.org TRNG test even better than that. It's just another visualization of how very badly FGU's physical engine fails.

A simpler and better visualization is in the first image that demonstrates how FGU's physical D8 ebb and flow within only 200 rolls. That's why I put it first and the other one last.

https://www.fourmilab.ch/random/
https://pracrand.sourceforge.net/
https://cristianopi.altervista.org/RaBiGeTe_MT/

https://en.wikipedia.org/wiki/Randomness_tests

I mean it would be nice when just every technical term is shortly introduced with its definition, to avoid that one needs to go through the links to understand that :) So, basically just an introduction of what you did and what the values represent, and how to read that, not just listing the result like Fail or Success :) That would certainly improve readability for people who are not experienced with all these technical terms, basically it would be good when the reader can interpret the result on one's own (now one simply needs to trust the Fail message). I understand though when that's annoying to do, because you did it in your free time etc. :)

Weissrolf

November 29th, 2020, 01:58

Weissrolf

November 29th, 2020, 02:00

And again, what really matters is the comparison to better (FGU /die) and proper (PCG, random.org) working RNGs. Even if we don't understand a thing about all the results we can easily see that one fails hard where others succeed.

I specifically did not include specific FGU /die results in the discussion yet, because physical 3D dice fails so much worse that we need to concentrate on them first.

smelton

November 29th, 2020, 02:25

I did far more work on this than I should have done,

Indeed. You compiled all this data comparing apples to oranges and then came here to conclude that not only are apples not oranges, there is something wrong with our apple because is it not an orange and we should get rid of it ... :confused:

Weissrolf

November 29th, 2020, 02:30

For comparison: PCG32 is the least complex form of the PCG family of random number generators and can be implemented with only 10 lines of code. Creating 10 gb of random data takes 26 seconds, corresponding to over 10 billion D256(-1) rolls. It succeeds with a test sequence length of 8 gigabytes where FGU's D8 already fails with a length of 8 kilobytes.

length= 8 gigabytes (2^33 bytes), time= 379 seconds
no anomalies in 1545 test result(s)

FGU's /die 256-1 fails at 64 kilobytes in comparison (and builds up suspicion before that).

Weissrolf

November 29th, 2020, 02:35

Indeed. You compiled all this data comparing apples to oranges and then came here to conclude that not only are apples not oranges, there is something wrong with our apple because is it not an orange and we should get rid of it ... :confused:

I compiled 375000 bytes of random data created via FGU D8 rolls and compared that to 375000 bytes of random data created by other RNGs (including FGU's /die). What exactly is it you like to contribute to the discussion by your snarky remark?

But let's just forget about the comparison: I sent FGU's rolls through well established randomness test applications and its D8 rolls fail easily and early. Why would you not want to get rid of such a very bad RNG?

And no, with a period of ebbs and flows within just 200 D8 rolls it is not even good enough for our usage scenario.

Kelrugem

November 29th, 2020, 02:46

It's rather impossible to do. We are not scientists here who even understand what this all means beyond fail and success. There just is no way to give a "short introduction". These are well established randomness tests and all we can do is trust their results, just like scientists around the world trust them and test them. Everyone wanting to go deeper needs to read through manuals and web-sites of scientists/coders who create and test random number generators (and even their disputes in papers and forums), and possibly study mathematics.

The first bitmap image is the most down-to-earth representation of FGU's PRNG failure I can give you. It is pure visualization of the numbers FGU rolled as black/white dots and should display random noise. Instead it displays visibly repeated patterns. That must not happen!

I did far more work on this than I should have done, but this represents a real number-crunching examination of FGU's physical dice results, free from any "I rolled too many crits today" bias. Not psychology, but mathematics. Does it go over my own head? Sure, but that's why Practrand tries to categorize it in readable fails and suspicions for dummies like us. ;)

No worries, I do not expect that you give proofs of all that stuff and how it works :) Let me clarify, I am a scientist (my field is sadly not statistics etc. sadly, would it have been about curved Yang-Mills-Higgs gauge theories or something like that then I could be more specific :D), and it is not just important to derive results but also to convince others of your results; especially the latter can be more difficult than deriving a result :) Of course there is trust needed when scientists and non-scientists speak with each other :) But when presenting results, one needs to think about who is the target audience and, based on that, you adjust how you present the result :) When I want to explain non-experts, what the Higgs boson is, I will not present the Yang-Mills-Higgs Lagrangian for example :) Do you know what I mean?

So, what I asked is just basically: What did you do actually? Your answer to LordEntrails gave some information I would have liked to see in the first post: So, sample size of the d8 rolls (an estimation of), and then what you did with them. Something like "I used these results to insert them into this programme which measures randomness", so, just some explanation of what you actually did; you do not need to explain what all of this means, but you directly started with the results and I had no idea what you actually were doing and were all of this came from :) I felt immediately lost. I needed to read your answer to LordEntrails to actually understand what you were doing and what the first graphic was about :) All the informations which non-experts cannot understand I would have simply omitted, just saying "when using that randomness-checker, I got a Fail as result", maybe uploading your d8 results (probably you have saved them?), such that someone else could check, too, if needed :) Especially SmiteWorks themselves for example :)

So, after reading your answers I now know what I would have liked to know in the first post already :) (to avoid that people get lost directly)

But of course I do not demand anything of that, just a comment about how it reads :) EDIT: In some essence, you could just copy parts of your answers here into the first post for readability :)

LordEntrails

November 29th, 2020, 02:52

If it says "Fail" in too many tests then it's not random.
I get that part :)

Sorry, but I really don't understand the question?! What range do you mean?
What I meant was something like this, out of 200 d8 rolls, my non-scientific, engineer only, brain would expect results like this;
Roll of 1 had 25 occurrences.
2 had 23
3 had 26
4 had 26
5 had 27
6 had 24
7 had 25
8 had 24

Something like that would be one thing (and maybe close to random?) but something like this;
1 had 5 occurrences
2 had 50
3 had 7
4 had 35
5 had 11
...

Would be very far from expected random results.

What I did was to create binary data out of the dice rolls where each byte consist of numbers 0-255. Since FGU's physical dice cannot roll D256(-1) I rolled D8(-1) instead, stripped the five 00000 high bits and formed one long bitstream of 375000 bytes (=1 mio rolls of D8). Each D8's binary representation is between 000 to 111, 2.67 D8 make one byte between 00000000 to 11111111. For comparison I did the very same with random.org rolls of 0-7.
Is that a methodology from one of the sites or is it something you came up with? I have no idea, and have no idea how to figure out, is it a valid methodology?

I do know other folks have claimed to have run analysis of chatlog.html rolls (actual campaign data) and have always come back and said that the results was within expected distributions. Now I have no idea their methodology(ies) or such. But it might be worth seeing if someone can find those old posts and analyses.

The first image is a bitmap where each binary 0 is black and each binary 1 is white. I found out that there is a period of only 600 pixels/bits where you see the ebb and flow of dark and light areas repeat in the bitmap image. 600 bits = 75 bytes x 2.67 = 200 rolls of D8. The image does not tell us which results were rolled, but it tells us that there are periodic ups and downs of more 1 bits vs. more 0 bits in the stream of consecutive rolls.

Again, not everyone has to understand what this all means, all you need to see is that both PCG and random.org look like random noise (the kind of static noise you may remember from old TVs), but FGU's D8 rolls show a clear periodic pattern.
The graphics are interesting, and with this explanation it makes sense now. Thanks.

smelton

November 29th, 2020, 02:53

Weissrolf

November 29th, 2020, 03:20

So, what I asked is just basically: What did you do actually?
Ah, of course, this I can do. I left all the explanation out, because if no one at SW was interested in the result and people come „pfff“ and „Blabla“ at me again then by do even more work?!

I am in bed now, but can write that down for you tomorrow after finally playing chess with my daughter (she kept asking).

I felt immediately lost.
I thought the visual comparison of FGU creating patterns where others create (TV like) noise was strong and clear that it already proved the main point. I was wrong in that.

maybe uploading your d8 results (probably you have saved them?),
I did, but how much more can one do with 1 mio. D8 rolls than what I did and no one here seems to have done before? I decided to offer these only per request, especially after being somewhat ridiculed for doing so many rolls. I also omitted results like average and such, because they were unsuspicious.

There are more randomness tests out there that are harder to use (even more so on Windows) and don‘t seem to produce better result.

Diehard
Dieharder
NIST
TestU01

Weissrolf

November 29th, 2020, 03:23

It's not a snarky remark .. more like a perfect analogy to summarize this thread. You are comparing a physics simulation (apple) to RNGs (oranges). No RNG is used for the 3D dice. If you want RNG, use /die or write an extension using your preferred RNG algorithm.
So the physics engine was never meant produce random results? Because I compare results, not engines.

/die cannot be used for standard dice (d2/3/4/6/8/10/20/100) or click based automation.

And the „write an extension“ arguments gets repeated far too often around here. I pay for finished products so that I do not have to do it by myself. This includes adventure paths and extensions.

Kelrugem

November 29th, 2020, 03:30

Ah, of course, this I can do. I left all the explanation out, because if no one at SW was interested in the result and people come „pfff“ and „Blabla“ at me again then by do even more work?!

I am in bed now, but can write that down for you tomorrow after finally playing chess with my daughter (she kept asking).

I thought the visual comparison of FGU creating patterns where others create (TV like) noise was strong and clear that it already proved the main point. I was wrong in that.

I did, but how much more can one do with 1 mio. D8 rolls than what I did and no one here seems to have done before? I decided to offer these only per request, especially after being somewhat ridiculed for doing so many rolls. I also omitted results like average and such, because they were unsuspicious.

There are more randomness tests out there that are harder to use (even more so on Windows) and don‘t seem to produce better result.

Diehard
Dieharder
NIST
TestU01

Cool, thanks :) (when it does not cause too much stress; I appreciate that someone had the energy to do all of that :) )

The problem I had with your first graphic was that I didn't see that it is a gif :D I just read the random.org at its bottom, and I thought you just put some example image of some wiki page as a reference :D Hence, I didn't realize that there are other images in that part, too :D

smelton

November 29th, 2020, 04:01

So the physics engine was never meant produce random results?

No actually. It's physics (an approximation at the least).

smelton

November 29th, 2020, 04:03

Because I compare results, not engines.

If you want to really test this out, find another app that uses physics based dice or pick up a d8 and throw it in a dice tray and compare that.

LordEntrails

November 29th, 2020, 04:46

I thought you might be interested in this post by one of the original devs; https://www.fantasygrounds.com/forums/showthread.php?10202-d20-does-not-appear-to-be-fair&p=75559&viewfull=1#post75559

smelton

November 29th, 2020, 05:49

I thought you might be interested in this post by one of the original devs; https://www.fantasygrounds.com/forums/showthread.php?10202-d20-does-not-appear-to-be-fair&p=75559&viewfull=1#post75559

That's similar to FGU ... variable input (start position, velocity, etc..) comes from the user and the result comes from the face of the die facing up after physics (collision, gravity, drag, etc..) does its thing, not a RNG algorithm.

Kelrugem

November 29th, 2020, 06:14

as far as I understood Weissrolf actually used the /die d8 command to do that (they had an extension automatically triggering that maybe, at least there where posts about that), so, the user based input is not contained here. I doubt Weissrolf did that many rolls all manually :D And it makes sense that the /die command gets studied since most dice commands are triggered by the character sheet :) (so, just pressing buttons, hence, also no user-based input with respect to start position etc.; when I understood everything correctly)

Weissrolf

November 29th, 2020, 09:29

If you want to really test this out, find another app that uses physics based dice or pick up a d8 and throw it in a dice tray and compare that.
But with all due respect, we are not interested in testing different physics engines at this point, we are interested in random die results. FGU's die results are not random, it's been proven by established methods of randomness testing and for the physical engine can even easily be made visible.

Of course we are happy to test and discuss new implementations of the engine that are meant to fix the results, but until these are offered I strongly advice to turn the broken thing off, either forcefully or optionally. Currently users have no means to do that.

"So the physics engine was never meant produce random results?"

No actually.

... variable input (start position, velocity, etc..) comes from the user and the result comes from the face of the die facing up after physics (collision, gravity, drag, etc..) does its thing, not a RNG algorithm.

This entropy is lost when dice are thrown with a script, but for all practical purposes this should make no difference in how random the results are.
I just proved that the results are not random but a deterministic ebb and flow and Smelton stated that the physics engine was never meant to produce random results. No one of my players is using their mouse for "variable input", we all use a double-click and automation (like automated save rolls).

So "for all practical purposes" FGU's physical die engine is broken and thus cheating.

DevildrummerX

November 29th, 2020, 10:06

Going to try and say this nicely, but why does this matter in the grand scheme of things? Personally I care more about the Dev's focusing resources on adding new features and patching bugs/glitches that prevent users from having an enjoyable experience playing the game. I really don't see why they should disable the entire physical dice rolling system because it does not meet the exact parameters of a true PRNG. Fantasy Grounds is a VTT first and foremost not a PRNG, its works great as a VTT with the added bonus of allowing for tons of useful automation. The fact the dice rolling system is not a true PRNG does not affect the performance of the program as a VTT or affect 99.9% of user's enjoyment of the system.

I just find it disrespectful that you think you speak for the vast majority of the program user base and suggest that the Dev's drop everything to disable a core system because its "not random enough."

Weissrolf

November 29th, 2020, 10:08

The problem I had with your first graphic was that I didn't see that it is a gif :D I just read the random.org at its bottom, and I thought you just put some example image of some wiki page as a reference :D Hence, I didn't realize that there are other images in that part, too :D
Sorry, this is absolutely my fault then. I did anticipate that for some users the GIF might not be animated or they miss the animation and thus put the FGU 200 rolls image first in the sequence, but seemingly you instead just saw the last image.

The original post was edited now to put a non animated image of FGU's results right beside the animation for more clarity and better comparison.

Zacchaeus

November 29th, 2020, 10:31

DevildrummerX

November 29th, 2020, 10:52

Can someone set out, in simple terms such that someone who hates Maths and does not understand a word of this thread, can get a handle on what is being proposed by the OP (other than simply saying the dice are broken). I'd like to try and understand why that conclusion has been reached and also why the current implementation has any effect on the games we play.

OP is comparing the physics based physical dice rolling engine to the non-physical "die" command in FGU and concluding that based on the results of rolling a D8 I think 375,000 times using each system, that the physics based engine is not random "enough" to be considered a PRNG. As Smelton has pointed out I personally believe the methodology used for the testing is flawed since the physics based system attempts to replicate physically rolling a D8 versus the /die command that uses a random number generator that does not have the added variables that undoubtable must be considered when using a physics simulation.

To add a bit more information. The OP is stating that when mapped out the result set of the 375,000 rolls in the physics based system contains patterns that are not present in the non-physics based system or other PRNG systems available for comparison.

As I stated I do not see why the Developers should disable or redesign the physics based rolling system because it is not meeting the definition of a random number generator. Personally I feel this thread is misleading since smelton has clarified that the Devs do not expect the results of the physics based system to fall within the same threshold as the non-physics based system.

Ludd_G

November 29th, 2020, 11:09

Weissrolf

November 29th, 2020, 11:12

Going to try and say this nicely, but why does this matter in the grand scheme of things?
It breaks everything connected to automation and dice rolling in general.

that prevent users from having an enjoyable experience playing the game.
If the game deterministically messes with your die rolls then playing the game can very much be less enjoyable. Game-systems like DnD5E and Pathfinder 2 mechanically put a *lot* of emphasize on the die result.

I just find it disrespectful that you think you speak for the vast majority of the program user base
No disrespect to the user base meant and I definitively do not (think to) speak for the vast majority. I put these things to discussion...

and suggest that the Dev's drop everything to disable a core system
.. and (strongly) suggest to replace a broken core mechanic with an already present better working one until the bugs are fixed.

... because its "not random enough."
3D dice are not "not random enough", /die are "not random enough". 3D physical dice don't seem to be random at all. You decide how much it matters for you, but currently our group cannot decide for themselves, because there is no option to turn it off.

DevildrummerX

November 29th, 2020, 11:21

It breaks everything connected to automation and dice rolling in general.

If the game deterministically messes with your die rolls then playing the game can very much be less enjoyable. Game-systems like DnD5E and Pathfinder 2 mechanically put a *lot* of emphasize on the die result.

No disrespect to the user base meant and I definitively do not (think to) speak for the vast majority. I put these things to discussion...

.. and (strongly) suggest to replace a broken core mechanic with an already present better working one until the bugs are fixed.

3D dice are not "not random enough", /die are "not random enough". 3D physical dice don't seem to be random at all. You decide how much it matters for you, but currently our group cannot decide for themselves, because there is no option to turn it off.

How do you know it is "broken." Seriously explain how a physics based system not matching up with a non-physics based system takes anything away from the enjoyment of using FGU for a VTT? You are suggesting that the Devs waste the time to disable a core feature because you and your users have convinced yourselves that the game "may" be "cheating" you because it "may" be deterministically affecting your dice rolls? Are you intimately aware of how the Dev's are using Unity's physics engine to create a physics based simulation of the dice rolling? Are you aware of the amount of variables they are telling the physics engine to take into account? I'm assuming the answers are no. So again, how do you conclude the the system is broken based on a test the Dev's seem to feel is flawed from the start, and also how can you reach a valid conclusion when you do not know the exact setup of the physics simulation?

Again, Based on what Smelton posted eariler it appears that the Devs do not expect the physics based rolls to be comparable with the non-physics based system.

DevildrummerX

November 29th, 2020, 11:26

Does this not come down to whether there is sufficient variation in the 3D roll when triggered by double clicking on the dice box, as opposed to actually dragging and 'throwing' the dice using the mouse? If the 3D dice results are acceptably 'random' when 'thrown' is that carried over to rolls triggered by a double click, which would, indeed (to my totally unqualified eyes), seem to include much less physical variables? Is that the nub of the argument?

Frankly I'm not sure. Its hard to speculate much less draw objective conclusions about a physics based simulation without knowing how the simulation has been setup to begin with. The Dev's are the only one's that could possibly know how the simulation is being done. Honestly I'm not sure this discussion even belongs in the support section at this point. On first glance smelton did not seem to believe the two systems should even be compared to each other so I really don't see how this qualifies as a technical flaw in the program at this point.

Weissrolf

November 29th, 2020, 11:30

Can someone set out, in simple terms such that someone who hates Maths and does not understand a word of this thread, can get a handle on what is being proposed by the OP (other than simply saying the dice are broken).
- OP expected 3D dice thrown by double-clicks and automation to be random, as such randomness was stated in the past. OP even expected 3D rolls to be superior in randomness to /die rolls.

- Developer "Dupre" stated that 3D dice thrown by double-clicks and automation ought to be just as random as dice thrown by mouse-movement.

- OP proved that 3D dice thrown by double-clicks are not random, but follow a strong periodic distribution. OP further proves that /die rolls (while flawed) are much more random than 3D rolls.

- OP suggest to fix the bug leading to non-random 3D rolls and advises to even turn off the 3D dice engine in exchange for the /die engine until then. OP also suggest an option for users to turn 3D dice off in the future and to replace the current /die engine with a better one.

- Developer "smelton" denies OP's assumption that 3D dice were ever meant to create random results.

- OP is at a loss.

I'd like to try and understand why that conclusion has been reached and also why the current implementation has any effect on the games we play.
The current implementation does not create nearly sufficient random results. Everyone can draw their own conclusion about that, but currently no one can opt out of the worse die rolling mechanic.

Weissrolf

November 29th, 2020, 11:36

How do you know it is "broken."
It does not create random results.

On first glance smelton did not seem to believe the two systems should even be compared to each other so I really don't see how this qualifies as a technical flaw in the program at this point.
The reason why I specifically compared 3D dice to /die was to demonstrate that FGU is at least capable of producing much better random results. The reason for visually comparing 3D dice generally to PCG, random.org was to demonstrate what random results should look like, /die comes much closer to this.

You could just ignore any comparison to /die, my original statement that 3D dice rolling in FGU is broken because it does not create close to random results still stands. Of course that is build on the assumption that 3D dice are even meant to create random results, which we surprisingly just learned is not the case. Well, I was surprised, other may not have been.

Jiminimonka

November 29th, 2020, 11:37

How many dice rolls did you make manually?

I suggest you leave this topic alone, the dice are random enough (and the devs have logs the confirm their own assertion as has been stated several times already in other forum posts over the years, and over the past week or so). Go play D&D and play chess with your daughter instead of spending countless hours on dice rolling.

Also in Settings there is a Manual Dice entry Off/On option, turn it on and get the players to roll their own dice and give you the results.

EDIT: Sorry Weissrolf, I should not have said that.

Weissrolf

November 29th, 2020, 11:47

What I meant was something like this, out of 200 d8 rolls, my non-scientific, engineer only, brain would expect results like this;
I did not forget about your post, preparing an answer with useful numbers and graphs took some time and is not finished yet. I will post the results as soon as possible.

Is that a methodology from one of the sites or is it something you came up with? I have no idea, and have no idea how to figure out, is it a valid methodology?
All randomness tests either test RNGs directly (as in code needs to be implemented) or by testing the RNG results as binary data files. I had to come up with a method to convert FGU's text-chat results to binary data files.

I do know other folks have claimed to have run analysis of chatlog.html rolls (actual campaign data) and have always come back and said that the results was within expected distributions. Now I have no idea their methodology(ies) or such. But it might be worth seeing if someone can find those old posts and analyses.
If you take a look at the links I provided (just start at the Wikipedia page) you will see that these randomness test suites are much more advanced and calculation intensive that what we can come up with ourselves.

Just looking at means and very obvious distribution is a very basic method of testing for randomness. But since we just learned that FGU's 3D dice are by design not even meant to create random results it seems that all analysis efforts of the past were quite in vain.

Zacchaeus

November 29th, 2020, 11:48

Go play D&D and play chess with your daughter instead of spending countless hours on dice rolling.

No, please be respectful. The OP has put a lot of work into this and if he has a point it is valid work and valid to point it out on the forums. I have a reasonable understanding of probabilities and I can see his point - although I'm not sure how he got to it since I don't know what tools were used to get to the conclusion. Let's all just leave it as it is at the moment. The devs will no doubt have a look at the situation and make any adjustments or not that they see fit.

Weissrolf

November 29th, 2020, 11:55

How many dice rolls did you make manually?
Just as many as both my Pathfinder groups during play. None, we only use double-clicks and automation.

I suggest you leave this topic alone, the dice are random enough
No, they are not random enough, they are hardly random at all. And according to Smelton they are not even supposed to be by design.

(and the devs have logs the confirm their own assertion as has been stated several times already in other forum posts over the years, and over the past week or so).
A dev just stated that 3D dice rolling is not meant to produce random results and I posted proof that indeed it does not. Contrary to other forum posts over the years and week I used scientifically established methods to test for that randomness and I even provided a very basic visualization that still demonstrates periodically determined results.

Here is another one:

1st million D8:

https://i.imgur.com/LUnZYqu.png

2nd million D8:

https://i.imgur.com/h094tlS.png

3rd million D8:

https://i.imgur.com/qWjuCNC.png

Also in Settings there is a Manual Dice entry Off/On option, turn it on and get the players to roll their own dice and give you the results.
Yes, I know. When we play tomorrow I will tell my players about the very serious flaws of the 3D dice/automation engine and offer them to roll themselves. It's a chore, though.

DevildrummerX

November 29th, 2020, 11:56

This is literally a pointless discussion at the moment. As I stated, the Developers are the only ones who are going to be able to definitively know based on the design and implementation of the physics based dice rolling system whether the system is intended to provide a random result set that is comparable to a traditional RNG algorithm. I think it is safe to assume that the 3D dice rolling system in FGU is proprietary to Smiteworks, they alone know what standards/thresholds they used to test the randomness of the dice roll results. Without that context it is impossible for us to know whether the dice rolling system is working as intended.

At this point I will wait for MoonWizard to chime in with what information he feels is necessary to share regarding the expected behavior of the physics based rolling system.

Jiminimonka

November 29th, 2020, 11:57

No, please be respectful. The OP has put a lot of work into this and if he has a point it is valid work and valid to point it out on the forums. I have a reasonable understanding of probabilities and I can see his point - although I'm not sure how he got to it since I don't know what tools were used to get to the conclusion. Let's all just leave it as it is at the moment. The devs will no doubt have a look at the situation and make any adjustments or not that they see fit.

Yes, sorry I was out of order. Sorry Weissrolf.

Valyar

November 29th, 2020, 12:04

What is the impact of this issue, I would ask this first. All ruleseta are not using the 3D dice with gravity to generate the outcome. Or I misunderstood.

Ludd_G

November 29th, 2020, 12:23

Weissrolf

November 29th, 2020, 12:37

What is the impact of this issue, I would ask this first. All ruleseta are not using the 3D dice with gravity to generate the outcome. Or I misunderstood.
3D dice are used any time you click on a roll on your character sheet or when an automated roll is made (like save rolls for DND/Pathfinder). Even when you use /die with a standard dice size (d2/3/4/6/8/10/20/100) FGU automatically enforces 3D dice rolling.

Weissrolf

November 29th, 2020, 12:40

Would this work as an interim solution for those not happy with the way 3D dice may be working, but don't want to slow play down using manual dice? I don't know what it's mechanics are under the surface but it may be using /die?

https://www.fantasygrounds.com/forums/showthread.php?62047-Instant-Dice-Disable-Rolling-Animation
This extension might indeed offer a workaround, thanks for pointing to the extension. I will have to check its results to make sure that randomness is equal to /die (which has its own flaws not discussed here yet).

Weissrolf

November 29th, 2020, 12:52

Yes, sorry I was out of order. Sorry Weissrolf.
Gladly accepted, but it happens too often on this forum (and elsewhere). I specifically asked for on topic, open and polite discussions from the get-go, because my experiences here are tarnished in that regard.

- Numbers, graphs and results of (stress)tests can be discussed objectively.

- Controversial suggestions can be discussed subjectively, like me suggesting to temporarily disable 3D dice (when I assumed they were even supposed to be random).

I also suggest everyone to ask me questions directly instead of talking about me in the third person and asking for the thread to be closed and thus discussion to be smothered. You may not be comfortable with how I apply QA like stress-tests to FGU in order to reveal flaws in either implementation or design of its functions. But that's how you test if you want reproducible results with hard numbers instead of hearsay and confirmation bias.

Weissrolf

November 29th, 2020, 12:54

I did not forget about your post, preparing an answer with useful numbers and graphs took some time and is not finished yet. I will post the results as soon as possible.
I provided graphs for distribution (occurrences of 1 to 8) for 3x 1 million D8 rolls in this post: https://www.fantasygrounds.com/forums/showthread.php?64056-3D-dice-results-are-not-random&p=560567&viewfull=1#post560567

Here is the distribution after converting 1 million D8(-1) rolls into 375000 bytes of binary data (corresponding to D256-1 rolls):

https://i.imgur.com/2ugt30G.png

Feel free to ask any (specific) questions!

damned

November 29th, 2020, 14:27

No, they are not random enough, they are hardly random at all. And according to Smelton they are not even supposed to be by design.

A dev just stated that 3D dice rolling is not meant to produce random results and I posted proof that indeed it does not. Contrary to other forum posts over the years and week I used scientifically established methods to test for that randomness and I even provided a very basic visualization that still demonstrates periodically determined results.

Here is another one:

1st million D8:

https://i.imgur.com/LUnZYqu.png

2nd million D8:

https://i.imgur.com/h094tlS.png

3rd million D8:

https://i.imgur.com/qWjuCNC.png

I just rolled 100k d8s.
My results look similar to yours but are not the same (your three results are also similar but not the same).
https://www.fantasygrounds.com/forums/attachment.php?attachmentid=41514

#1 Rolled: 13151 Occurrence: 13.151%
#2 Rolled: 13386 Occurrence: 13.386%
#3 Rolled: 12221 Occurrence: 12.221%
#4 Rolled: 11792 Occurrence: 11.792%
#5 Rolled: 12859 Occurrence: 12.859%
#6 Rolled: 12907 Occurrence: 12.907%
#7 Rolled: 11526 Occurrence: 11.526%
#8 Rolled: 12158 Occurrence: 12.158%

In order #2, #1, #6, #5, #3, #8, #4, #7

Serious question here:
We are not dealing with numbers here. We are dealing with a representation of dice. Is the number that shows up on top the only measure of randomness? You have already suggested at least one other and that is runs of consecutive numbers. Ive asked before but dont recall a response - are there other measures of randomness? Where the dice stop on the table? What direction they are pointing? Are these relevant? Are there other measures you could do?

If you were rolling real physical dice I would expect to see some sort of approximate pattern emerge too. Most dice are not perfectly uniform, do not have exact angles on all corners, do not have exactly the same amount of material on each face, are not guaranteed to have identical weighting to each face, are not going to have identical wear on each face, on each edge etc.

On my rolls the odds of rolling 1-4 are 1.1% greater than rolling 5-8
The odds of 2d8 rolling 9 with 1, 2, 7, 8 are 1.7% greater than rolling 9 with 3, 4, 5, 6

These numbers obviously jar with your expectations.
For me these numbers are completely acceptable in range, in frequency, in averages, in runs.
I cannot see anyone determining in advance what the next roll is likely to be with any meaningful degree of certainty.

Im also curious about why you do d8-1 and why you convert that to 256-1
What is the rationale and how can you use d8-1 to extrapolate a meaningful range of 256 values?

4151341514

Weissrolf

November 29th, 2020, 14:34

I provided graphs for distribution (occurrences of 1 to 8) for 3x 1 million D8 rolls in this post: https://www.fantasygrounds.com/forums/showthread.php?64056-3D-dice-results-are-not-random&p=560567&viewfull=1#post560567

Here is the distribution after converting 1 million D8(-1) rolls into 375000 bytes of binary data (corresponding to D256-1 rolls):

https://i.imgur.com/2ugt30G.png
For comparison here is the same graph of random.org D8 rolls, using the same scale as the FGU graph:

https://i.imgur.com/IKzUMhm.png

smelton

November 29th, 2020, 14:36

- Developer "smelton" denies OP's assumption that 3D dice were ever meant to create random results.

Wait what? Where?

A dev just stated that 3D dice rolling is not meant to produce random results

Again eh?

Assuming you mean my response to this:

So the physics engine was never meant produce random results?

No actually. It's physics (an approximation at the least).

You said "physics engine" but you meant 3D dice? The physics engine (Nvidia PhysX) is meant to simulate physics - not produce random results. Just to be clear, the 3D dice are meant to simulate rolling dice on a real world table - neither use a RNG for the result. There are variables that affect the physics calculations inside the physics engine that affect the result. I don't know how else to explain that.

Weissrolf

November 29th, 2020, 15:37

My results look similar to yours but are not the same (your three results are also similar but not the same).
Your scale starts at 0, my scale starts much higher and thus looks more dramatic.

In order #2, #1, #6, #5, #3, #8, #4, #7
1nd mio. order: #1, #2, #5, #3, #6, #8, #4, #7
2rd mio. order: #2, #1, #3, #6, #5, #8, #4, #7
3st mio. order: #1, #2, #6, #5, #3, #8, #4, #7

#2, #1, ... #8, #4, #7
#1, #2, ... #8, #4, #7

#4 and #7 are underrepresented in all sets, the graphs overall look the same. And number of occurrences is a rather mundane measurement of randomness in that most PRNG should handle these well. So if a PRNG fails this (as FGU 3D does) then it's an additional testament to the PRNG's weakness.

We are not dealing with numbers here.
If we want to assess randomness of results then all we end-users outside the black-box can do is deal with numbers (=results). That being said, we just learned from Smelton that FGU's 3D dice are not even meant to produce random numbers. So if this statement is true then there is no basis for discussing its randomness weaknesses. You can't blame table designed to be square for not being round. I did not get the impression that this design choice was communicated that clearly in the past, though.

Is the number that shows up on top the only measure of randomness? You have already suggested at least one other and that is runs of consecutive numbers. Ive asked before but dont recall a response - are there other measures of randomness?
Yes, various, and these are thoroughly tested by the randomness test suites I applied (Practrand, RaBiGeTe, ENT). My first noise image demonstrates congregations of 0 bits and 1 bits in an ebb and flow within a 200 rolls period.

Where the dice stop on the table? What direction they are pointing? Are these relevant? Are there other measures you could do?
Currently I see FGU's 3D dice engine as neither emulating physical rolls well, nor resulting in random numbers. The former is mainly because in FGU dice don't turn while in the air, they mostly only turn on their very last hit on the ground unless they hit a chat-window border first. I saw dice hit the ground 2-3 times on one of their edges without them turning until the final settling movement.

You would have to awkwardly throw them in parallel to the desktop surface like this to get the a similar real dice experience. And then they still fly faster than most of FGU's double-click/automate roles or hardly turns, the latter of which can be observed in FGU. I guess this is kind of a Casino way of throwing (rather than rolling) dice, but hardly ever never see this at a RPG table.

https://i.imgur.com/lsTJSiF.png

If you were rolling real physical dice I would expect to see some sort of approximate pattern emerge too. Most dice are not perfectly uniform, do not have exact angles on all corners, do not have exactly the same amount of material on each face, are not guaranteed to have identical weighting to each face, are not going to have identical wear on each face, on each edge etc.
But with FGU the variance is not in dice, but in the randomly created impulse and consecutive physics emulation. One or both of these are weighted in a way that causes non-uniform and even periodic distribution of numbers. And the end result is so weak in randomness that it can hardly be called as such. Again, we just learned that randomness is not the goal of said 3D dice rolls, so anyone wanting to use 3D dice needs to consider this.

On my rolls the odds of rolling 1-4 are 1.1% greater than rolling 5-8
The odds of 2d8 rolling 9 with 1, 2, 7, 8 are 1.7% greater than rolling 9 with 3, 4, 5, 6
That only tells us that the average of all rolls is close enough to the theoretical mean. If this was off then again this would only underline the extraordinary weakness of the RNG, fortunately FGU 3D dice do not fail this very, very basic test.

Im also curious about why you do d8-1 and why you convert that to 256-1
Randomness test suites test binary data either on bits (0s and 1s) or bytes (0-255) basis. Originally I expected the physical dice engine to be based on the same RNG (original impulse of 3D dice when no mouse-movement is used). So I started by using /die 256-1 (0-255 = 1 byte) but then hit the now fixed bug that the highest /die number was *never* rolled. Then I read this:

No, non-physical dice and physical dice use completely different systems.
So I had to come up with a way to test 3D physical dice whose results had to finally be converted to a stream of bytes (file).

What is the rationale and how can you use d8-1 to extrapolate a meaningful range of 256 values?
The D8(-1) is the only physical die that fits completely into a set of bit: 0-7 = 000 - 111. Every other physical die does not fit into complete sets of bits: 0 - 20 = 0000 - 10100 (11111 = 31). When you convert the D8-1 rolls into binaries you get 00000000 - 00000111, so you strip the leading 00000 and then combine 2²/³ (2.67) bits into one byte. Or rather, you turn 8 x 3 rolls = 24 bits into 3 bytes and the whole series of roles into a single large bit/byte-stream. With random numbers you can do that, because:

If this is random: 011 010 011 000 000 110 010 011
than this is random: 01101001 10000001 10010011

And to make sure that my logic was correct I specifically got D8-1 rolls (integers of 0-7) from random.org for comparison that underwent the same transformation. The random.org rolls are random, as are a comparison of byte sized (D256-1) numbers done via PCG32 (mkrandom.exe).

Weissrolf

November 29th, 2020, 15:48

Wait what? Where?
...
Again eh?

You said "physics engine" but you meant 3D dice?
Look, I apologize for using a wrong term then, my definition in this sentence was too broad and English is not my native language. But the whole thread is very extremely clearly about 3D dice (called "physical dice" by Moon Wizard), so please excuse me for assuming you were not trying to just pin me down on semantics without adding something on topic.

It is hard to imagine that you mean this question even sincerely after all the data and unambiguous statements provided by me, but I will give you the benefit of the doubt. We both learned something about communication here.

The physics engine (Nvidia PhysX) is meant to simulate physics - not produce random results. Just to be clear, the 3D dice are meant to simulate rolling dice on a real world table - neither use a RNG for the result. There are variables that affect the physics calculations inside the physics engine that affect the result. I don't know how else to explain that.
I understand this and already understood this to begin with. On top of that I understand that when no mouse-movement is present to feed the engine with an external impulse (entropy) then a PRNG is used to determine initial impulse, rotation of each die and maybe some other parameters. From there a deterministic physics engine tries to emulate the real world more or less successfully. It does so in an oddly fashion, but as long as the end result is random I see this as eye-candy.

Now: FGU's 3D physical dice engine fails to deliver random results. Knowing how FGU's own /die engine is capable of far better results (albeit problematic on their own) there are two ways how this could have gone wrong. Either the underlying RNG mechanics used for initial parameters is bugged, or the physics engine is too deterministic in its outcome which may cause it to "blur" away initial randomness instead of keeping or improving on it. Both are bad.

Moon Wizard

November 29th, 2020, 16:48

While I appreciate the amount of work put into your project, we do not have any plans to change the 3D dice simulation used in Fantasy Grounds at this time. We do have some plans to tweak the dice rolling behaviors to have a little more rotational velocity to look better, but changing the underlying system is not in our plans.

If you do not feel that the engine is sufficiently random for your tastes, we do have a manual option so that you can roll physical dice or use an external system of your choice to input numbers that you feel are sufficiently random.

I will not be commenting on this thread further; because there are more important priorities to deal with at this time. (Especially given that we just launched less than a month ago.)

Regards,
JPG

fubeca150

November 29th, 2020, 17:20

Weissrolf

November 29th, 2020, 17:24

3D dice results are not random at all. This is different from being not "sufficiently random", especially if your own /die engine can do worlds better. How can you look at this and not call it a problem/bug?

https://i.imgur.com/az7ETZ5.png

Well, my players are not going to be happy about this and I do not want to force them to manually enter all dice results. Let's hope that the extensions linked to earlier in this thread works better than what FGU offers by default and force.

Jiminimonka

November 29th, 2020, 17:25

I noticed something. According to casinos and https://www.gamescience.com/ and Gary Gygax https://mxplx.com/Meme/2616/ the dice should have equally sized sides. The dice on the FGU table have rounded edges, like those cheap dice you get in the D&D box that are badly weighted.

Moon Wizard get a file and make sure those sides are all the same and the dice rolls will be more evenly distributed. :p Problem solved :D

Oh and there is a setting in FG that lets you enter dice rolls from actual dice rolls, so the players roll with their dice and enter the results manually.

EDIT: I think the Gygax link above is from the 1st edition of the AD&D Dungeon Master's Guide although I haven't looked in that book for over 30 years now and the link above is missing the graphics he provided.

eporrini

November 29th, 2020, 17:50

LordEntrails

November 29th, 2020, 18:00

Earlier a question was asked or implied, "Why does this matter?" And that led me to wonder, how do the FG 3D dice compare to real physical dice? Because in general most people feel that real dice are "random enough". So, Are real physical dice random? No, no they are not. They are not even close. Even Game Science dice are not random.

For one analysis, see; https://www.awesomedice.com/blogs/news/d20-dice-randomness-test-chessex-vs-gamescience

So, I think this is a fascinating discussion, but to me, until shown otherwise, it is just academic and does not have any meaningful impact to my game. My Chessex dice are random enough for my game, are they for you? Then until someone compares the FG results to the linked results and can show that FG dice are not close to real dice in randomness, I'm not going to be concerned.

Again, real dice are not random. That's ok with me.

Jiminimonka

November 29th, 2020, 18:01

This is an important topic for me and calls into question the integrity of the rolling system. I certainly want to feel rolls are random. I generally roll myself manually in the box, which I hope offsets this to a degree based on the thread. As a possible “simple” suggestion, can you alter/randomize (even after the roll starts) the top number position on the die? That way no matter the physics the starting number varies causing randomness in outputs?

Can the roll used when clicked be in any way randomized in direction or throw power?

According to Weissrolf it calls into question the integrity of the rolling system. But not according to the devs who wrote the rolling system.

The physics engine picks up the mouse movements are the "seed" for the dice rolling physics.

eporrini

November 29th, 2020, 18:03

What about when you click on a die to roll it?

LordEntrails

November 29th, 2020, 18:10

What about when you click on a die to roll it?
It uses a different random seed.

Weissrolf

November 29th, 2020, 18:44

My whole test procedure did not move the mouse and thus not externally seed the 3D dice engine. None of my players drags the dice, they all use buttons, as does any kind of automation in FG. According to an earlier developer it should not matter much anyway.

And there is a difference between rolling a whole bunch of imperfect real physical dice versus what FGU tries to simulate rather clunky. And how does that even compare to having every D8 in every FGU instance on every computer roll less 4s and 7s and always have a higher chance to roll 2-4 on 2D8 than than anything else.

3D dice results are plain broken to a point that's not even funny. In the binary data-file created out of D8 rolls there are 1225x 35 vs. 2495x 36, double as many 36! That's not distribution, that's determination!

Anyway, I am testing the "No dice animation" extension now, which may provide at least a much, much, much better RNG experience, even if still flawed.

PS: Implementing minimal PCG32 is 12 (twelve!) lines of C code, there are also libraries available in C and C++. And even the lowest PCG32 is already better than /die and maybe faster on top of that.

fubeca150

November 29th, 2020, 18:53

My two groups were having die roll results that were so same-y by most players double-clicking on the button over the last few weeks that it didn't seem like proper PRNG to me. This is something we only started noticing over the last month but I have no idea if that's just observation bias. So much so that half my group (including myself) read a bunch of white papers on PRNG methodology and such that I actually understood the OP.

We were trying to figure out how to embed a hardware RNG into Windows (turns out it's possible but takes some serious work) since hoping Smiteworks would add support for one seems like it would be a lost cause. Turns out it's really easy in Linux and I was starting to think about ways to run a virtual machine and get the hardware RNG down to the Linux VM.

To find out that FGU is using "Physical" 3d dice is a relief. Real dice are deterministic, but it's not very realistic for someone to recreate an exact throw. I'm okay with that being simulated in 3d dice.

My druthers would be to put more oomph into the initial throw vectors of the dice when double-clicking a skill or dropping the dice onto a target. I'm not a fan of dice falling limply to the table and maybe sometimes rolling a single facing. That's why I enjoy throwing the dice in the chat tray and see them vigorously bouncing around.

Weissrolf

November 29th, 2020, 20:53

Every D8 you ever buy and roll has the highest chances to roll a 1 or 2 and the lowest changes to roll a 4 or 7 and then 8. That is what clicked-rolled 3D dice currently do in FGU.

damned

November 29th, 2020, 21:23

Thanks for bringing this up. I've been tossing dice by dragging them into the tray and giving them a good shake. I've noticed much reduced randomness from the players that are clicking the skill button to roll and it keeps coming up "2" for the first half of the night. One players dice looked they kept getting dropped together and one kept landing on the other. To my untrained eye it looked like his physics dice weren't getting random rotation and force vector.

The takeaway is that the workaround is to toss them in the tray myself, which I like to do anyway. I'll mention this thread to him so that he can drag to the chat tray next time we play.

Please can you post your chat logs.
That way we can see the logs and the erroneous results.

If you throw an equal number of every die you might be surprised to note that 1 in 8 rolls will be a 2 (on average)

Almost 89% of results will be 10 or lower.
Over 80% will be 8 or lower.
Over 69% will be 6 or lower
Over 51% will be 4 or lower
and 1 in 8 rolls will be a 1

celestian

November 29th, 2020, 22:03

Kelrugem

November 29th, 2020, 23:28

Yeah, really nice and great information :) As a physicist and mathematician I may be biased (so, I am interested, I mean), but sadly my expertise is not with respect to probability theory, I could say something about the physical/philosophical idea of randomness (especially, since quantum physics one discusses until today whether there is true randomness in the real world), but here it is about how to test randomness coming from programmes and sadly I never had to face such type of matter such that I do sadly not know how to argue here.

That may be an advantage of Roll20: They connected their physical die engine to a real quantum physical experiment, probably leading to better random results. But LordEntrails argument is also worth looking at, because, indeed, the way real dice work and are used do not suggest that they are "perfectly" random, one approach could be to describe dice movement by classical mechanics (i.e. a deterministic framework), it is just not possible for us to calculate that explicitly, that is way too complicated, especially with the shuffle at the beginning. Hence, we use statistics to study dice systems, but just because the chosen mathematical framework is statistical, this does not imply that the "true nature" is random; statistics is also often chosen to simplify calculations (often in order to do any type of calculation which is otherwise not possible). What I want to say with that, is, that FG's approach may be therefore nearer to the "real dice nature" while Roll20's approach is nearer what we may want to call "true randomness" (as suggested by quantum physics). At least, for further discussions, one needs to fix what type of randomness we want and expect; and then the results can be tested against that :)

Edit: So, certainly interesting discussion, but when we drift into the "philosophy" behind what randomness means (as I did with my post :D) one can quickly run into a rabbit hole, because, as I said, the "nature of randomness" is discussed until today. Though when the physical die engine indeed shows a rhythmic pattern, I may concur with Weissrolf that this may suggest some "problem" (since symmetries like that can help to predict certain results as with non-ideal dice, which may stand against what we call "ideal die")

Weissrolf

November 30th, 2020, 00:01

As a physicist and mathematician you are far more qualified to understand and interpret the results than I am. I just started looking into the test software and problems around randomness testing after my last thread on that topic was locked, seeing how all those "confirmation bias" discussions led to no quantitative or qualitative results.

Take a look at how 3D dice "float" and fall in FGU versus how they fly out of your hand (or dice cup) onto the table. To me FGU's 3D dice neither resemble real nor ideal dice, both are far off. So going for the statistical/ideal approach seems both easier and more fair to everyone. Since weighted distribution is easier measurable than expected I may have a look at D20 distribution later on.

Currently I am in the process of collecting data/rolls done via the "Instant dice" extension. It's a much slower process, because I have to send mouse-clicks to FGU instead of chat input strings (/die commands) and because my memory gets eaten alive by FGU (just damaged my WoW installation in the process). A first quick test with over 100k D8 rolls looks promising.

That being said, I already mentioned that /die has its own problems, just far less pronounced than 3D. /die might still be sufficient, while 3D really seems broken. Here is a test on /die. This looks worse than it might be and other tests look much better than anything 3D can even hope to achieve.

length= 1 megabyte (2^20 bytes), time= 0.3 seconds
Test Name Raw Processed Evaluation
BRank(18):256(2) R= +3748 p~= 3e-1129 FAIL !!!!!!!!
BRank(18):384(1) R= +5405 p~= 3e-1628 FAIL !!!!!!!!
BRank(18):512(1) R= +8161 p~= 1e-2457 FAIL !!!!!!!!
[Low1/8]BRank(18):256(1) R= +2650 p~= 9.8e-799 FAIL !!!!!!!
[Low1/16]BRank(18):256(1) R= +2650 p~= 9.8e-799 FAIL !!!!!!!
[Low4/16]BRank(18):256(2) R= +3748 p~= 3e-1129 FAIL !!!!!!!!
[Low4/16]BRank(18):384(1) R= +5405 p~= 3e-1628 FAIL !!!!!!!!
[Low4/32]BRank(18):256(1) R= +2650 p~= 9.8e-799 FAIL !!!!!!!
[Low4/64]BRank(18):256(1) R= +2650 p~= 9.8e-799 FAIL !!!!!!!
[Low8/32]BRank(18):256(2) R= +3748 p~= 3e-1129 FAIL !!!!!!!!
[Low8/32]BRank(18):384(1) R= +5405 p~= 3e-1628 FAIL !!!!!!!!
[Low8/64]BRank(18):256(1) R= +2650 p~= 9.8e-799 FAIL !!!!!!!

Failures on BRank suggest that in some way the PRNG output, or at least part of it, was extremely linear, producable strictly by xoring bits of previous PRNG output. This is classically seen on LFSRs / GFSRs, but sometimes it shows up on PRNGs that are *supposed* to be chaotic, but actually have some component that isn't really.
Are likely better, maybe even faster and still easy to implement alternatives out there? It sure seems so.

Weissrolf

November 30th, 2020, 00:14

PS: When we go down the rabbit's hole of quantum physics we can also start to discuss that there really is no physical equivalent of the "ideal Pi" and round circles. At some point we reach the Planck length and thus reality's display resolution starts to get blurry - or rather "uncertain" - at best. Sometimes ideal mathematics and physical reality are at odds with one another. ;)

Weissrolf

November 30th, 2020, 00:30

This is some great data.

Indeed. You compiled all this data comparing apples to oranges and then came here to conclude that not only are apples not oranges, there is something wrong with our apple because is it not an orange and we should get rid of it ...

This kinda information is very handy for improving FG.

While I appreciate the amount of work put into your project, we do not have any plans to change the 3D dice simulation used in Fantasy Grounds at this time. We do have some plans to tweak the dice rolling behaviors to have a little more rotational velocity to look better, but changing the underlying system is not in our plans.

Thanks for taking the time to analyze it and show what is going on.
Thanks for the flowers, it's well appreciated! :)

We play tomorrow and I plan on getting the "Instant Dice" extension ready by then. Once I analyzed its output (likely equal to /die) I will likely suggest to everyone using click- and automation-rolling to do the same.

celestian

November 30th, 2020, 02:26

Thanks for the flowers, it's well appreciated! :)

We play tomorrow and I plan on getting the "Instant Dice" extension ready by then. Once I analyzed its output (likely equal to /die) I will likely suggest to everyone using click- and automation-rolling to do the same.

Look forward to checking out the extension.

Kelrugem

November 30th, 2020, 02:52

Weissrolf

December 1st, 2020, 00:38

3D dice, 1 mio. rolls, distribution of D8 results:

#1 13,01%
#2 13,07%
#3 12,90%
#4 11,41%
#5 12,86%
#6 12,90%
#7 11,21%
#8 12,65%
https://i.imgur.com/tF2oY0Y.png

https://i.imgur.com/QhcA4Ho.png

Instant dice extension, 1 mio. rolls, distribution of D8 results:

#1 12,41%
#2 12,40%
#3 12,43%
#4 12,55%
#5 12,61%
#6 12,58%
#7 12,52%
#8 12,50%
https://i.imgur.com/ZRW0zI2.png

https://i.imgur.com/bNemlrB.png

Weissrolf

December 1st, 2020, 00:55

Some fodder for thoughts:

#2 13,07%
vs.
#7 11,21%

This may look like only a 1.86% difference, but consider again. Yes, there is "only" a 1.86% higher chance to roll a #2 vs. a #7, but this translates to 16.6% more total #2 results than #7 results (130.700 vs. 112.100).

Compare that to the same rolls done via Instant Dice extension:

#5 12,61%
vs.
#2 12,40%

A 0.21% higher chance to roll a #5 vs. a #2. This translates to 1.7% more total #5 results than #2 results (126.100 vs. 124.400).

Weissrolf

December 1st, 2020, 01:52

3D dice, distribution of byte values (0-255)

https://i.imgur.com/ca8wbTh.png

Instance Dice extension, distribution of byte values (0-255)

https://i.imgur.com/aRkkak4.png

Asterionaisien

December 1st, 2020, 12:21

3D dice, distribution of byte values (0-255)

https://i.imgur.com/ca8wbTh.png

Instance Dice extension, distribution of byte values (0-255)

https://i.imgur.com/aRkkak4.png

Greetings Mr. Weissrolf,
there is a way to know the distribution of d20?
Considering the fact that d20 is a die used in a lot of the most played games, this become an important issue.
The extension solves this for all practical issues, but could be interesting to know how this affect the games (I'm thinking especially of the 5ed adv/disadvantage mechanic).
Maybe a good solution for SW to solve the issue could be to let physical rolling remain as a graphic effect( so one can mantain the fun of rolling dice) but the results are determined by the above extension code?
My best thanks.

Weissrolf

December 1st, 2020, 13:11

Well, the extension (or /die for that matter) are not really good random generators, 3D is just so broken that in comparison /die shines brightly and might suffice for the time being. I am not sure yet if /die and the extension create comparable results. Some measurements suggest differences, but then I am of course testing different sets of rolls (1 mio. D8 each).

Using the extension seems to be leaps and bounds better than using 3D dice, though, so I told all my players to disable 3D dice. Of course that did not keep my Ranger player from rolling 4 rolls below 5 in a row. But I told him that even if he keeps missing all 4 rolls per turn at least Fantasy Grounds is not cheating on him on top of that. :D

I am currently in the process of collecting 1mio. 3D rolls of D20. We can demonstrate distribution from that, but we cannot send it through established randomness tests, because 0-19 does not fit into full sets of bits. I will post the distribution in 2 or 3 hours.

Neovirtus

December 1st, 2020, 14:54

Hi Weissrolf. Interesting stuff. As other people here have mentioned, it is very difficult for anyone to understand the depth of what you've done here. I think we can all see that it is interesting, and that there is likely an issue that you have highlighted, but your explanation of the process you used is a bit hard to follow (not trying to suggest fault - this is a forum post, not a white paper! Just communicating my thoughts to you). Like others here, I am a scientist though no statistician. Have you considered putting the raw data up somewhere for download? I would likely play with it, and I'm sure others would as well.

While I want to wrap my brain around some of the methods you've used, perhaps you can answer this simple question first - what exactly is being shown in this image?

https://i.imgur.com/0Yyhs5P.png

In the GIF you illustrate that these oscillations are roughly on a period of about 200 rolls. What is the y-axis? I assume the x-axis is # of rolls (and that this is a small subset of your total data set, owing to the fact that you show the "200 roll" scale bar, and I know you did millions of tests). What does the dark band actually indicate? Are you coloring the pixel on a grayscale where black is 1 on a d8, and white is 8 on a d8? Please elaborate.

Also, I think you have focused on the fact that certain numbers come up more frequently than others, which I agree is an issue... but to my mind and as many people have mentioned, no die is completely perfect. And though I agree these should be, and are intended to be, it probably wouldn't bother me too much. However, the issue that is more problematic to me is the thought that there are periods of time where rolls are much more likely to be low than others.

Weissrolf

December 1st, 2020, 15:51

3D dice, 1 mio. rolls, distribution of D8 results:

#1 46177 4,62
#2 49035 4,90
#3 54112 5,41
#4 50356 5,04
#5 54087 5,41
#6 50882 5,09
#7 48173 4,82
#8 49519 4,95
#9 48401 4,84
#10 49038 4,90
#11 48341 4,83
#12 49441 4,94
#13 49499 4,95
#14 47458 4,75
#15 50715 5,07
#16 54863 5,49
#17 50399 5,04
#18 53634 5,36
#19 49337 4,93
#20 46533 4,65

#16: 5.49%
vs.
#1: 4.62%

There are 18.8% more #16 than #1 results (54863 vs. 46177).

https://i.imgur.com/GnkK7sw.png
It will take time to get either Instant Dice extension or Random.org distribution for comparison. I expect both to be much better, though.

Weissrolf

December 1st, 2020, 16:17

While I want to wrap my brain around some of the methods you've used, perhaps you can answer this simple question first - what exactly is being shown in this image?
1 mio. D8-1 (0-7) rolls in a bitstream of 0 and 1, every 0 being black and every 1 being white. 3 bits/pixels = 1 roll, 600 pixels = 200 rolls. The image I posted is downscaled to 25% of the original image, which helps visualization. The original image is 1800 x 1667 px = 3.000.600 px / 3 = 1.000.200 px. The 200 extra pixels are empty, to fill a complete square image.

In the GIF you illustrate that these oscillations are roughly on a period of about 200 rolls. What is the y-axis? I assume the x-axis is # of rolls
There are no x and y axis. The whole image is a stream of rolls, but I found out that when I limit the x axis to multiples of 600 bits/pixels (=200 D8) then this pattern emerges. It helps visualize the periodic nature of roll results.

What does the dark band actually indicate?
The dark bands indicate result numbers consisting of more zeros (0) in their binary/bit representation, 7 = 111, 0 = 000, 4 = 010, 3=011. There are also bright bands in between the dark bands with results consisting of more ones (1) than zeros. When you increase contrast on that image it will turn into pure black/white stripes instead of blotches.

Are you coloring the pixel on a grayscale where black is 1 on a d8, and white is 8 on a d8? Please elaborate.
No, the grayscale is a product of downsampling the original b/w image to 25% size. Since every D8-1 result consists of 3 bits/pixels the bands do not represent high vs. low results, but rather certain numbers (with more or less 1 vs. 0) clutching together.

Here is the original image at full size. Make sure to open it outside of your browser, because else you very likely see it resamples using bilinear interpolation.

41568

but to my mind and as many people have mentioned, no die is completely perfect.
FGU's 3D dice are quite completely "perfect" in that they always favor the same set of numbers over other numbers. And c'mon, how is FGU's physics emulation of dice flying and rolling anywhere close to reality? Currently we get the worst of both worlds, physics and RNG. And you often see player exchange dice at a table (for whatever effect).

To clarify one thing: When I post result distributions of 3D dice here then these distributions are specific to the chat window size of my system (2560 x 1600px display using default chat window size). In this configuration FGU's 3D dice always oscillate between these two states:

https://i.imgur.com/WJxhzNo.png

It is quite likely that another chat window size will lead to another pre-determined distribution of results. Look at this for comparison:

https://i.imgur.com/4kCf1OS.png

Neovirtus

December 1st, 2020, 16:55

Interesting, thanks for the expanded explanations.

FGU's 3D dice are quite completely "perfect" in that they always favor the same set of numbers over other numbers. And c'mon, how is FGU's physics emulation of dice flying and rolling anywhere close to reality? Currently we get the worst of both worlds, physics and RNG. And you often see player exchange dice at a table (for whatever effect).

I'm not arguing that. I was merely trying to express that the less than perfect distribution of numbers is less an issue to me than the periodicity. I think people are much less likely to notice a 0.5% difference in frequency of a single result than they are a trend in results (periods of low or high numbers being more frequent).

The dark bands indicate result numbers consisting of more zeros (0) in their binary/bit representation, 7 = 111, 0 = 000, 4 = 010, 3=011

So, for your d8-1 bitstream
1 = 000
2 = 001
3 = 010
4 = 011
5 = 100
6 = 101
7 = 110
8 = 111

Either way, the dark period in your chart indicate a concentration of 0's. Indicating a higher frequency of 1, 2, 3 and 5 occurrences. If that's true that is definitely interesting.

Weissrolf

December 1st, 2020, 17:13

I think people are much less likely to notice a 0.5% difference in frequency of a single result than they are a trend in results (periods of low or high numbers being more frequent).
It's more of a 1.5% absolute difference, which results in over 16% relative difference. Quite large actually.

Here is an emphasized version using a median noise filter (radius 4).

FGU 3D dice

https://i.imgur.com/eQvGoEA.png

FGU Instant Dice extension

https://i.imgur.com/EjLuhfX.png

Random.org

https://i.imgur.com/70jzyLy.png

Like I wrote before, I do not consider these results "imperfections", but a plain broken/bugged state.

Neovirtus

December 1st, 2020, 17:41

Like I wrote before, I do not consider these results "imperfections", but a plain broken/bugged state.

And you are welcome to that opinion, I was merely stating mine.

Weissrolf

December 1st, 2020, 17:56

I was not questioning that, sorry if it "sounded" that way. ;)

But FGU's 3D dice are far beyond real dice "imperfections", those dark band distributions should not happen with even bad random generator (/die isn't so shiny either). And real dice are thrown very differently (more variable!) to how FGU does it, so real imperfections lead to different results compared to FGU's engine. Not to mention that every single real die is weighted differently, too, while in FGU they are all the same.

eporrini

December 1st, 2020, 17:59

Jiminimonka

December 1st, 2020, 18:12

I am by no means requiring perfection in the randomness of rolling, but clearly this really needs to be addressed. I don’t care about physics vs random generation, only that rolls are “reasonably random”. Moon Wizard has indicated this will not be addressed, which is concerning. My players (and I) LIKE seeing rolls, so simply using the extension would feel bad and take away from the game. For me the randomness of /die is “sufficiently” random.

A couple of questions and a suggestion.

Questions:

Do these ratios apply to the “seeded” rolls being made by dragging dice and virtually throwing them in the box?

Do these ratios apply to drag/drop roles that apparently generate some kind of “seeding” when u lift a die?

Suggestion: Can we brainstorm fixes that solve this while allowing rolling (both in the box as well as drag/drop or click) to continue to be used? Preferably something relatively easy to implement for the dev team?

Moon Wizard said that is doesn't need attention because its not broken.

My players have streaks of good rolls and streaks of bad rolls. And that also happens when you are sitting at a table with real dice.

Humans used to think the world was flat too (and that is making a come back recently).

Also dice rolls are not totally random which is why casino dice that are made one special machines so each side is exactly the same dimensions as all the others so the dice rolls are more equal. Eventually they get discarded (is it 10,000 rolls?) and sold off. D&D dice are not as precise nor do they ever have to be.

I'm unsubbing to this thread because its kinda doing my head in.

Weissrolf

December 1st, 2020, 18:29

Do these ratios apply to the “seeded” rolls being made by dragging dice and virtually throwing them in the box?
Since I had to collect data for a very high number of rolls it was not possible to seed them using mouse-movement. My players click-role and any ruleset automation also do not use mouse-movements.

Do these ratios apply to drag/drop roles that apparently generate some kind of “seeding” when u lift a die?
I don't know about the lifting part, but I read that you have to generate a minimum of mouse-movement for it to be used for rolling. If you move too slow then a fixed minimum is used (or maybe a RNG minimum). My results only apply to click-roles and automation, which always use random/minimum movements.

Weissrolf

December 1st, 2020, 18:31

Edit: I fixed the Instant Dice extension to use a proper seed. It used the same C library Rand() function that /die should be using. This means that it uses different implementations on each OS (Windows/OS X/Linux).

Weissrolf

December 1st, 2020, 19:48

I am currently in the process of editing the "Instant Dice" extension. It currently used the math.random function to create random numbers and depends on the system it is being run on. I may either include a changing seed or try to implement a better PRNG to begin with. Since I am not experienced in coding LUA this may take some tries.

Weissrolf

December 2nd, 2020, 20:35

3D dice, 1 mio. rolls, distribution of D20 results:

#16: 5.49%
vs.
#1: 4.62%

There are 18.8% more #16 than #1 results (54863 vs. 46177).

https://i.imgur.com/GnkK7sw.png
It will take time to get either Instant Dice extension or Random.org distribution for comparison. I expect both to be much better, though.
Instant Dice, 1 mio. rolls, distribution of D20 results:

#6: 5.05%
vs.
#7: 4.98%

There are 1.8% more #6 than #7 results (50453 vs. 49631).

https://i.imgur.com/QrDoX93.png

eporrini

December 2nd, 2020, 21:03

If another D20 1m was rolled using 3D is the expectation it would gravitate into the same pattern as the prior 1m?

Does anyone think this can be fixed just by adding some kind of additional seeding to the auto and click rolls?

Weissrolf

December 2nd, 2020, 21:15

I did 3x 1 mio. D8 rolls and they all showed the same pattern with very little variation. Currently I do not plan to do another mio. D20. 3D click-/auto-rolling is bad and my energy currently goes into finding better workarounds (after fixing the Instant Dice extension to make it usable).

Auto-rolls and click-rolls should not be using 3D dice, because they lack the random mouse-movement seed. And even then I am not convinced that this isn't more of a physics engine problem than random seed problem.

Best solution would be to update FGU to a more modern version of LUA, which introduced better RNG (random.new, later also implemented in math.random).

Jiminimonka

December 2nd, 2020, 21:17

Weissrolf

December 2nd, 2020, 21:29

You are stating the obvious here.

Integrating a new version of LUA could introduce compatibility issues if any old functions were deprecated. LUA v5.1 was introduced 2006 and ended 2012.

Weissrolf

December 3rd, 2020, 12:09

I found pure Lua implementations of three different random number generators. Unfortunately I cannot implement them, because FG's Lua API excludes "setmetatable" which is used by said implementation.

Next I may compare FGC's 3D dice results with FGU's, because they do roll visibly different.

We do have some plans to tweak the dice rolling behaviors to have a little more rotational velocity to look better, but changing the underlying system is not in our plans.
I specifically wonder how can having "a little more rotational velocity" only make it look better? Should that not also change the possible outcome? Currently 3D hardly ever turn, often only just before they settle. Sometimes they don't even turn when they hit the table several times. So more rotation should surely change that?

Weissrolf

December 3rd, 2020, 12:18

FYI: 3D dice fail the PractRand randomness test with 8 KB test sequences (range of numbers tested), on my W10 system Instant Dice fails with a 128 KB test sequence. Both are small and testament to how bad these RNGs are, but the latter at least survives larger test sequences than the former.

Weissrolf

December 3rd, 2020, 16:22

Instant Dice, 1 mio. rolls, distribution of D20 results:

#6: 5.05%
vs.
#7: 4.98%

There are 1.8% more #6 than #7 results (50453 vs. 49631).

https://i.imgur.com/QrDoX93.png
FG Classic, 1 mio. rolls, distribution of D20 results:

#2: 5.21%
vs.
#10: 4.80%

There are 8.6% more #2 than #10 results (52099 vs. 48002).

https://i.imgur.com/pJ1I4Uo.png

FG Unity in direct comparison:

https://i.imgur.com/IzQrufY.png

There are 18.8% more #16 than #1 results (54863 vs. 46177).

Weissrolf

December 3rd, 2020, 16:41

Distribution of Doubles (2 dice consecutively showing the same result) in percent (expected value ~0.25% per die side):

https://i.imgur.com/fEqtkK3.png

In terms of DnD5E "advantage" rolls this means: Your chance of rolling another 3 after an initial 3 are 33% higher than rolling another 1 after an initial 1. And on average your chances of rolling a double 3 are 16.5% higher than they should be.

For comparison:

https://i.imgur.com/lW0LRTd.png

https://i.imgur.com/UwLwHIw.png

Jiminimonka

December 3rd, 2020, 18:09

Trenloe

December 3rd, 2020, 18:34

I want to see the dice table have different surface qualities and possible coffee stains, both sugary and non-sugary and see how the dice interact on those surfaces and while we are at it add VR dice and possible table shake.
If you don't have anything constructive to contribute then please don't post anything. Please leave this thread to those who are interested in this type of theoretical stuff.

Weissrolf

December 3rd, 2020, 18:52

I want to see the dice table have different surface qualities and possible coffee stains, both sugary and non-sugary and see how the dice interact on those surfaces and while we are at it add VR dice and possible table shake.
This would be a nice touch (pun intended), but only useful if the initial click-/automation-roll is anywhere close to how users roll dice via mouse-movement. And even then the physics engine needs to be good enough to not just blur everything into much too deterministic paths (literally).

Neovirtus

December 3rd, 2020, 20:02

Based on this thread I told my players about the potential issue at the beginning of last night's session and recommended them to use the manual mouse roll to truly randomize the initial vector of the roll (if they were concerned about the kind of imbalances you have calculated). I was particularly concerned that one player who had been having bad luck was going to instantly blame all his bad rolls on this...

Turns out he had been rolling with the mouse the whole time and didn't realize he could just double click to roll, lol. So fortunately no great strife resulted at my table.

Weissrolf

December 3rd, 2020, 20:11

Frankly, at this point I fully expect manual mouse-movement to just lead to the very same distributions that I measured for click-rolling. Click-rolls use random vectors to initialize the 3D roll and unless FGU uses an external library specifically for this they use the very same math.random (C rand) RNG that is used in the Instant Dice extension (or /die).

So either the initial random vector creation is messed up in FGU (worse than FGC) or the whole 3D physics engine results in originally more random vectors becoming less random. Both are possible, but I wouldn't wonder if it was the latter, especially seeing how Classic seems to do better than Unity.

At least the Instant Dice extension allows to turn off 3D dice for click/automation-rolls while leaving drag-rolls intact. That's essentially what I was asking to be implemented as an option in the first post.

When you want less 1 and 20 results on D20 rolls, though, then better keep using 3D (click-)rolls. ;)

Neovirtus

December 3rd, 2020, 20:29

Hmmm, seeing as the roll is completely physics based, I assumed the distribution you saw would be tied to the only RNG aspect of that process, namely the initial random vector. I figure manually generating that initial vector would only improve things. As I've mentioned previously in the thread, I'm less concerned with the the distribution of results being off than I am with there being trends in the results. If nothing else, manual rolling of physical dice should limit trends I would think.

Weissrolf

December 3rd, 2020, 21:09

If the random vector is based off the same RNG as /die and Instant Dice (math.random = C rand) then either creation of said vector is flawed or the physics processing afterwards. Classic looks quite a lot better and judging from the visible dice movements the physics are different, the RNG seems to be the very same.

It's worth mentioning that math.random = C rand is differently implemented on each platform (Windows, OS X, Linux). Especially OS X is said to suffer from its implementation (based/equal BSD).

eporrini

December 3rd, 2020, 21:10

I hope the Smiteworks team decides to take this seriously. Not having some level of integrity with rolling is a significant and foundational game issue. I am torn with using the instant dice extension as no rolling animation at all takes away from the joy of rolling on the virtual tabletop. I am leaning towards asking my players to roll manually whenever possible for now, even though I know certain number of rolls will get triggered through click or automatically.

I am re-petitioning the developers to work towards a real solution for this ASAP as a priority over other enhancements. I feel this is significant enough that I would use it as a differentiator when deciding on what virtual tabletop I would want to use to play on. The physics don't need to add up to "perfectly random", but we need something dependable enough that we don't feel like we are playing a rigged game with weighted dice.

Weissrolf

December 3rd, 2020, 22:22

Instant Dice only removes click-rolls and automatic-rolls. You can still drag every control onto the chat or tokens to do a 3D roll. So it's kind of best of both worlds. Make sure to download my mod, though, else it rolls the same results every time you restart FG.

LordEntrails

December 3rd, 2020, 22:30

I'm still waiting for a comparison to physical dice. We know physical dice are not random, or anywhere near random, so until their is info that FGs dice are more or less random than physical dice, I don't personally have enough info to be worried about it.

The other point mentioned by Weissroff just a few points ago is that the math.random function is implemented differently based on platform, so curious how these comparisons change per platform.

Also, from what I can tell, all these results were created by a script and a as yet unproven methodology. Admittedly I don't understand much of what has been posted here, but I still don't know how repeatable these results are in a play situation versus script generation where the rolls are created at regular intervals.

Finally, I just don't see this having a big effect on my game, but understand that others might see it differently.

Jiminimonka

December 3rd, 2020, 22:44

Trenloe

December 3rd, 2020, 22:52

I feel this is significant enough that I would use it as a differentiator when deciding on what virtual tabletop I would want to use to play on.
It can only be used as a differentiator if this is truly a big enough issues for you and your players, and if *exactly* the same tests are done on the VTTs you’re comparing. Otherwise it can’t be a differentiator if you can’t tell the difference.

The physics don't need to add up to "perfectly random", but we need something dependable enough that we don't feel like we are playing a rigged game with weighted dice.
You’re the second person to use this type of language in this thread. How does this mean you’re playing a rigged game or you’re getting cheated? Does FG pick a bad number for your roll just to cheat you or rig the result specifically to someone’s detriment or advantage? No, it doesn’t.

I think some perspective is needed here. Weissrolf has done an interesting statistical analysis. It’s very theoretical and doesn’t simulate the timing and spread of different dice rolls within an actual gaming session. Even if it did, the results don’t indicate anything regarding being cheated or playing in a rigged game.

Weissrolf

December 3rd, 2020, 23:00

I'm still waiting for a comparison to physical dice.
No one can give you that, it's just not possible in a practical way. But I gave you a comparison to Classic's physics engine, which creates different and arguably better results to Unity's engine.

We know physical dice are not random, or anywhere near random,
We know that we all have a whole satchel full of dice that all behave differently and none of them move in the air and on the table like they do in FGU.

In FGU dice are always weighted the same. My chance to throw a 3 instead of a 20 are 16% higher in FGU and my chance to roll double 3s instead of double 20s are 37% higher. Does every of my physical D20 in my satchel follow that exact same pattern at every table ?

so until their is info that FGs dice are more or less random than physical dice, I don't personally have enough info to be worried about it.
Given the many years of discussion and "confirmation bias" arguments that seem to have happened in the past it does seem like other users are worried, though.

The other point mentioned by Weissroff just a few points ago is that the math.random function is implemented differently based on platform, so curious how these comparisons change per platform.
They likely won't. I already proved that my Windows 10 based math.random results are far superior to 3D dice results on the same computer, I do have access to OS X and Linux, but shy away from the additional work.

Click-rolls and automation-rolls are always based on a random seed, which is always prone to the same limitations as /die and math.random. Unless FGU uses a superior library or Lua based RNG for 3D dice randomness, that is. Would FGU allow Lua's "setmetatable" then we implement (copy & paste) free MIT license based RNGs ourselves. Or FGU could make use of a much better RNG than math.random (C rand) by using one of the free libraries out there.

Also, from what I can tell, all these results were created by a script and a as yet unproven methodology.
No, they were not. Autohotkey was used to either type "/die d20{enter}" into FG's chat window or to click on a character sheet button. Then FG's chatlog file was parsed (much of it manually) and converted to formats suitable for scientifically established randomness tests and Excel (for graphs and further calculations).

how repeatable these results are in a play situation versus script generation where the rolls are created at regular intervals.
Unless FG initiates a new seed with every roll it does not matter how long the intervals are and either random is random or it is not. Only rolls matter. I did try to use a new seed for every roll, but this was not possible in (FG's) Lua, only one seed per restart/reload could be set for math.random.

Finally, I just don't see this having a big effect on my game, but understand that others might see it differently.
Switching from Classic to Unity lowered my chances to roll a 1 or 20 by 10%. Sure makes for a less "extreme" gaming experience (crits & fumbles).

seycyrus

December 3rd, 2020, 23:27

...
Suggestion: Can we brainstorm fixes that solve this while allowing rolling (both in the box as well as drag/drop or click) to continue to be used? Preferably something relatively easy to implement for the dev team?

The best fix in the short term (and maybe even the long term) is to use some sort of totally random agency that lies outside of the framework of FG to initiate rolls by both players and NPCs. It would be even better if the external agency consisted of multiple random agencies interacting in a random fashion.

We could call it Heuristic Users Manipulating Almost Normal Statistics.

Weissrolf

December 3rd, 2020, 23:44

I am not sure if this is meant as ridicule or as lighthearted joke?!

seycyrus

December 3rd, 2020, 23:52

I am not sure if this is meant as ridicule or as lighthearted joke?!

It is true, and it is clever - the best of both worlds.

eporrini

December 4th, 2020, 01:59

Significant statistical data has been provided by Weissrolf regarding the percentages of rolls and issues currently in place. My concern is that we can all accept RNG and bad rolling when we know the foundation is in fact "random" and defend it as (just what it is), randomness. Once we've determined it's not randomness at the core, but biased rolling, how then do we defend a bad night or issues with rolls? My biggest worry is that Weissrolf or someone will actually take this further and invalidate the manual drag/drop rolling as well, which would be a GIANT bummer. I think sufficient efforts have been made here to have the developers (at a minimum) respond and review this and provide a determination and next steps as to the severity and possible fixes. Maybe this is a non-issue, but the dialog has been completely one sided so far with Weiss providing significant findings and Smiteworks saying we're not looking at that. Believe me I understand there are a TON of things the team is working on. I see updates weekly with fixes and features being implemented. For me, the CORE of the game needs to work foundationally and we can then stack whatever else we want on top of it.

1. Let's improve the randomness/3d rolling ASAP or debunk the need to do so.
2. Just calling out (different thread) the LOS (for me) is now broken as well and needs fixing, it's why my entire group moved off of FDC to FGU. I get "NO ONE" is going to come to a happy compromise on this however it's implemented, so some kind of toggle to help set LOS will be needed to allow flexibility there.

The rest of whatever is in the devops pipeline can wait.

damned

December 4th, 2020, 03:38

I have also run many hundreds of thousands of rolls and while I have not done any hard statistical analysis on them other than frequency of numbers.

My results lean towards those of Weissrolfs in terms of frequency, though not identically.

However I have not ever seen in my casual browsing of the hundreds of thousands of results, nor in my 9 years of using the software, anything that I can point to and say - that is not possible/probable.

The numbers of rolls we are talking about here in these threads are massive. In all the years that people have talked about bad dice I cannot recall any of them uploading their chat logs so that we can see the actual events as opposed to what they recall as having happened. There is no rocket science involved in sharing of logs, you dont have to write any scripts or throw a million dice and do exhaustive analysis on it. Just access the campaign folder on the GM computer and upload the chatlog file. Im genuinely interested to see peoples logs. Please if you have a bad night on the dice share the logs.

Ive mentioned on this thread that if you threw a bucket of equal numbers of all the different dice sizes (4/6/8/10/12/20) that 50% of the time the number will be in the range of 1-4. People do remember low numbers/rolls and statistically you are going to see lots more low numbers than high numbers because of the different sizes of the dice.

Now in saying all that - I think that sufficient replication of long range odds has been shown that it would be worth the devs having another look at the issue.

And counter to that - in my opinion - there has not been anything that shows even remotely that in terms of the numbers of dice thrown in any session or campaign there is anything that makes me feel the dice are not sufficiently random so if changing this in any meaningful way is difficult I personally would accept that as an answer.

damned

December 4th, 2020, 04:03

As an example - in terms of variance and tolerances - here is an example of the tolerances that are and were deemed acceptable between what your car speedometer says and how fast you are actually driving.

https://www.racq.com.au/cars-and-driving/safety-on-the-road/driving-safely/speedo-accuracy

Australian Design Rule 18 sets out the accuracy standards for vehicle speedos.

Until July 2006 this rule specified an accuracy of +/- 10 percent of the vehicle’s true speed when the vehicle was travelling above 40km/h.

That is, at a true vehicle speed of 100km/h the speedo could indicate between 90km/h and 110km/h.

An odometer accuracy of +/- 4 percent was also a requirement.

From July 1 2006 a new standard began its phase in and by 1 July 2007 all new vehicles had to comply. The new standard requires that:

The speedo must not indicate a speed less than the vehicle’s true speed or a speed greater than the vehicle’s true speed by an amount more than 10 percent plus 4 km/h.
Odometer accuracy is no longer defined.

What this means:
For a vehicle travelling at a true speed of 100km/h, the speedo must read between 100km/h and 114km/h. The effect of this is that many drivers will find that at 100km/h they are driving up to 14km/h below the speed limit if they rely on the vehicle’s speedo.
The speedo must always read 'safe', meaning the vehicle must not travel faster than the speed indicated by the speedo.
This change was made to align Australian vehicle rules with those already in place in Europe. It applies to all Australian motor vehicles except mopeds.
Dealers will generally not attempt to correct speedo error unless it exceeds the legal requirements.

In real life almost everything is variable and not exactly what it is stated as. And we dont care so long as for all practicable purposes it works.

Jiminimonka

December 4th, 2020, 08:44

Jiminimonka

December 4th, 2020, 08:55

Casino dice are manufactured to much stricter standards than D&D polyhedral die (https://www.midwestgamesupply.com/dice_manufacturing.htm) but even they are not predictable in their results.

I conclude that the dice in FGU are fine and not broken.

Weissrolf

December 4th, 2020, 09:11

The speedo must always read 'safe', meaning the vehicle must not travel faster than the speed indicated by the speedo.
There is a practical and technical reason for a car's speedometer tolerance: Different tires have different diameters/rolling circumference and speedometer measurements are based on RPM of the wheels.

https://tiresize.com/speedometer-calibration/

Does our little computer program suffer from such real-world challenges? No, it does not.

In real life almost everything is variable and not exactly what it is stated as. And we dont care so long as for all practicable purposes it works.
I emphasize this again: Just by switching from Classic to Unity my chances to roll a 1 or 20 dropped by 10%.

Weissrolf

December 4th, 2020, 09:15

I conclude that the dice in FGU are fine and not broken.
So Classic's dice are broken then, because they behave consistently different?! Which worlds' physics are we playing in when we roll dice in FGC vs. FGU? Again, my chances to roll a 1 or 20 dropped by 10% just by switching from Classic to Unity.

Trenloe

December 4th, 2020, 09:54

I emphasize this again: Just by switching from Classic to Unity my chances to roll a 1 or 20 dropped by 10%.
That’s misleading - 10% of what? 10% suggests that it’s going to happen 1 in 10 times. For a 1 the drop was approx 5,000 rolls out of 1,000,000 rolls - 0.5% of the total rolls, 1 in 200 times. So, statistically, if you roll a d20 200 times in a session one of those rolls that wasn’t a 1 should be a 1, and one of these rolls that wasn’t a 20 should be a 20, but we don’t know what roll result we’d need to change.

But, as FGC is above the mean for a 1 on a d20, the actual FGU deviation from the mean (each result should occur 50,000 times in 1,000,000 rolls) and taking your results from post #96: 50,000 - 46,177 = 3,823 rolls less than the mean. This results in a roll of 1 on a d20 not happening when it should, statistically, 1 time in 261 rolls. For most of my games (and I play a d20 based system most often) that wouldn’t even be once per session.

Like I said, this is an interesting theoretical presentation of data but needs some perspective on the actual impact to our games.

Jiminimonka

December 4th, 2020, 10:03

I'm going to upload my chat log tonight. And next session on Monday. And especially when players think the rolls are bad....

Weissrolf

December 4th, 2020, 10:37

That’s misleading - 10% of what?
10% lower chance to roll a 1 or 20 in Unity relative to Classic, consistently. While Classic tends for a more "extreme" game, Unity tends for a less "extreme" game.

So, statistically, if you roll a d20 200 times in a session one of those rolls that wasn’t a 1 should be a 1, and one of these rolls that wasn’t a 20 should be a 20, but we don’t know what roll result we’d need to change.
Averaged over time you are correct. But every single D20 roll in FGU has a 7.5% lower chance to hit a 20 than it statistically should have, consistently. And every double roll has a 17% lower chance to hit double 20s, consistently. My dice are all weighted and I cannot grab into my satchel to get another one other than using the Instant Dice extension.

But, as FGC is above the mean for a 1 on a d20
...For most of my games (and I play a d20 based system most often) that wouldn’t even be once per session.
Again, this may be correct over averaged period of time, but every single die roll has a 2.5% higher chance to hit 20 than it should have. That being said, for the D20 the deviation is 3 times smaller in Classic than it is in Unity, the latter of which I consider more seriously bugged for the time being (not having checked Classic at detail).

Like I said, this is an interesting theoretical presentation of data but needs some perspective on the actual impact to our games.
The perspective is that every single roll is affected even when it does not hit you over the head while you play games. It may even be that someone playing on OS X has consistently differently weighted dice than you, so either of you is always getting lower chances for a D20 than the other player. It may not bother many of the veterans around here, but at least people should be informed and get a chance to switch it off.

I am a GM and every time my players roll sh*te someone in the group starts explaining how computers cannot roll proper random numbers. Frankly, I would like to get that discussion out of my games, especially in the middle of the action. There are easy to implement statistically good PRNG algorithms out there and we are using something that's over 20 years old and then put an even less random system on top of it just for the eye candy.

Weissrolf

December 4th, 2020, 10:41

I'm going to upload my chat log tonight. And next session on Monday. And especially when players think the rolls are bad....
FGU's rolling would have to be even much more blatantly weighted for a human to see any pattern in a log full of different dice being rolled. I am already surprised about the level of weighting present, but it likely is not so bad that you could see it by looking hard at log files. The only kind of human pattern recognition you can hope for is from the bit based noise images I provided.

Trenloe

December 4th, 2020, 10:45

So, I keep saying that we need to have perspective and look at how this relates to our actual gaming experience.

A while back, Doug did some campaign d20 roll analysis based off GMs submitting their chat log, info in this thread: https://www.fantasygrounds.com/forums/showthread.php?37768-Campaign-Game-Statistics

Let's look at the first chatlog analysed - from post #2 in that thread. The chatlog shows 9 gaming sessions - a couple of those are shorter than others, so let's go with 8 gaming sessions. The results are shown here: https://www.fantasygrounds.com/calendar/gamestats.php?id=2503

Specifically, let's look at the d20 results: https://www.fantasygrounds.com/calendar/gamestats.php?id=2503&category=d20+Rolls Here is that data presented in Excel:

https://www.fantasygrounds.com/forums/attachment.php?attachmentid=41643

Keep in mind this is for FG Classic. For a direct comparisson we need to look at the FGC data here: https://www.fantasygrounds.com/forums/showthread.php?64056-3D-dice-results-are-not-random&p=561913&viewfull=1#post561913 From the two specific pieces of data supplied: #2: 5.21% vs. #10: 4.80% for 1,000,000 rolls, we see that across 8 gaming sessions, with 976 rolls, a 2 occurred 5.02% of the time and a 10 occurred 5.12%. Looking at rough data from the graph, we'd expect this dastardly #1 roll to occur 5.1% of the time - but it's actually 4.4% of the time - well low in this game for FGC (and also well below the statistical analysis of the #1 result in 1,000,000 simulated rolls for FGU).

I'm sure if you looked at other real-game statistics (available in that thread) you'd see differing results. See here for example with a lot more rolls: https://www.fantasygrounds.com/calendar/gamestats.php?id=1737&category=d20+Rolls or here for a single session example with not so many rolls: https://www.fantasygrounds.com/calendar/gamestats.php?id=2506&category=d20+Rolls

So, what can we take from this? In a real-game situation the statistics of simulated 1,000,000 d20 rolls don't apply. Sorry, Weissrolf, but that's simply the cold facts. And most people who deal in statistics would have foreseen this - because simulating something that happens a few hundred times across a few hours by rolling 1,000,000 times gives statistics, but doesn't represent real (virtual) world experience.

Can we please put to bed the notion that FGU is broken and cheating and playing a rigged game with weighted dice? The statistics are interesting, but as the above actual game chat log analyses show, statistics like this don't directly cross over to real-game experiences. And the devs certainly should *not* drop everything to try to address something that has little-to-no impact on our actual gaming experience.

Jiminimonka

December 4th, 2020, 10:58

I am a GM and every time my players roll sh*te someone in the group starts explaining how computers cannot roll proper random numbers. Frankly, I would like to get that discussion out of my games, especially in the middle of the action. There are easy to implement statistically good PRNG algorithms out there and we are using something that's over 20 years old and then put an even less random system on top of it just for the eye candy.

And thus we come to the point. Humans blaming their dice rolls on some external source - bad dice, bad luck, bad software. As always humans looking for something to blame.

I have had sessions where the orcs shooting arrows at 1 target missed 15 times in a row... and the player didnt complain at all. Those orcs sure did though.

Save your chat logs and upload them here next time the players complain.

Trenloe

December 4th, 2020, 11:06

Averaged over time you are correct.

Again, this may be correct over averaged period of time...
These statements contradict the data you're presenting. Rolling 1,000,000 d20s and presenting the data for those 1,000,000 rolls together is "averaged over time". All of the statements you make like "7.5% lower chance to hit a 20" and "every single die roll has a 2.5% higher chance to hit 20 than it should have" is based off the "averaged over time" data you've collected.

Anyway, please see my "real game" post above. I'm moving on now, this thread is nothing more than a bunch of interesting statistics. Post #125 above shows that these type of statistics don't directly correlate to an in-game experience.

Weissrolf

December 4th, 2020, 11:18

So, I keep saying that we need to have perspective and look at how this relates to our actual gaming experience.
I do not disagree.

A while back, Doug did some campaign d20 roll analysis based off GMs submitting their chat log...
While again I agree that taking data out of real games can be helpful, we have to take three things into consideration here:

1) The data provided does not seem reliable. One of your linked logs lists results of 0 (zero) on a D20 and the other one lists a 22 on a D20. So let's agree that if we make use of another database then we need to make sure that we don't get completely wrong results. ("Five is right out."). I already noticed this a few weeks back when I first read and posted in Doug's thread.

I fully expect you to scrutinize the data presented by me when you add to the discussion. Everyone can only win by working together and revealing flaws in others data, so that it can be corrected.

2) While the data represents real gaming sessions the data basis is rather (too) small to even do statistical analysis on it. I did not do 1 mio. rolls because I ever thought this would represent how many times we roll per game or even per campaign, I did so to get statistically meaningful results and a more consistent look into the black box.

Chance is "the level of possibility that something will happen", it still might not happen at all for a long time. This brings us to:

3) Analysis of my data suggests a periodicity that we have yet to make tangible and find out how it affects our real games. While all those 1 mio. rolls statistics are not meant to represent periods of real game-time those 200 D8 rolls are more in the ballpark of "real". Unfortunately other dice than D8 cannot be converted into bit-streams so easily (if at all?), so we lack a better understanding of possible D6, D20 or D100 periodicity.

And locking at chat-logs likely will not reveal such, because looks less like periods of consistently low vs. high rolls, but more periods of consistently rolling certain numbers over other numbers. You could argue that it's not relevant then, but personally I wouldn't like my players being stuck in a period of numbers that deterministically make or break their rolls.

And here comes a solution like the "Instant Dice" extension to the rescue, because with all its flaws (C rand) it still behaves so much more random that we fight less dragons when we have to deal with it. It's a crude workaround, though, and I would like for FG's physics engine to do much better than it currently does, both statistically and visually (get rid of the "floating" dice, we are not playing on the moon, despite using "Lua").

Weissrolf

December 4th, 2020, 11:20

And thus we come to the point. Humans blaming their dice rolls on some external source - bad dice, bad luck, bad software. As always humans looking for something to blame.
Except that with 3D click/automation rolls those humans are correct, they are weighted.

Trenloe

December 4th, 2020, 11:26

2) While the data represents real gaming sessions the data basis is rather (too) small to even do statistical analysis on it.
I agree. But what I'm trying to get across is that statistical analysis is interesting - but it does not translate to real-game experiences. We don't roll thousands of dice a session, so averages over time data that has deviations of only a few percent doesn't apply to any noticeable impact on our games. If you look at multiple sets of data from real world sessions (I've given the link to a few above) you see the statistics are all over the place.

This proves that the "oh woe is me, FG is broken and needs fixing NOW!" train of this thread, based of purely statistics, does not translate into any impact on a real gaming session. To me, and I'd imagine most people who use FG to play RPGs with their friends, that is all that matters.

Keep going with the statistics, if that's what floats your boat, but please stop the claims the FG is broken and needs to be fixed. Because, for all gaming intents and purposes, it isn't.

Weissrolf

December 4th, 2020, 11:32

These statements contradict the data you're presenting. Rolling 1,000,000 d20s and presenting the data for those 1,000,000 rolls together is "averaged over time". All of the statements you make like "7.5% lower chance to hit a 20" and "every single die roll has a 2.5% higher chance to hit 20 than it should have" is based off the "averaged over time" data you've collected.
Yes, but averaged over a large and statistically strong period of time, not over a small and thus statistically weak period of time. Large data represents chance, small data represents anecdote.

Anyway, please see my "real game" post above.
Please see my answer.

I'm moving on now, this thread is nothing more than a bunch of interesting statistics.
We are talking about a virtual world existing in a digital domain, statistics are physical laws in there.

"Probability is a mysterious energy field created by dice that binds the virtual galaxy together. Harnessing the power of probability gives the Jedi, the Sith, and others sensitive to this statistical energy extraordinary abilities, such as levitating objects, tricking minds, and seeing things before they happen. While probability can grant users powerful abilities, it also directs their actions. And it has a will of its own, which both scholars and mystics have spent millennia seeking to understand."

Weissrolf

December 4th, 2020, 11:38

Keep going with the statistics, if that's what floats your boat, but please stop the claims the FG is broken and needs to be fixed. Because, for all gaming intents and purposes, it isn't.
FGU's 3D click/automation rolls are not even close to random. They fail tests with as little as 8000 bytes of data, start to look suspicious with 2000 bytes and very suspicious with 4000 bytes. The already bad math.random needs 16 times as many bytes to fail.

Again, implementing free and strong pseudo-random number generators can be done with as little as 12 lines of C code. There are also free Lua (5.1) based RNGs out there that can be copied and pasted. Very obviously 3D dice rolling was changed between Classic and Unity, so this opportunity could have been used to improve it instead of making it worse.

Trenloe

December 4th, 2020, 11:42

Large data represents chance, small data represents anecdote.
I'd argue that small data represents chance - you don't know what's going to happen next. Large data (massively large data) is where the statistics truly apply.

Maybe there is a game impacting pattern in FGU (like what you found with the seed issue in the LUA script you were looking at) that impacts the actual limited number of rolls made in an actual game. But the statistics presented in this thread so far don't support that. Until you can take the data from a lot of FGU chat logs and prove that in individual games there is a reproducible pattern across real-games situations (not simulating those situations) then everything you've presented in this thread proves little as to any actual gaming impact.

Weissrolf

December 4th, 2020, 11:55

200 rolls.

Trenloe

December 4th, 2020, 12:08

200 rolls.
You have no proof that this occurs in a real world game. This is based off 1,000,000 rolls which is, like you said, "averaged over time" so it is NOT real world data. And the real world data presented above doesn't support that the statistics apply to an actual FG gaming session (or series of sessions for that matter).

Sorry to "shout" with bold, but this is what you keep missing - your statistics aren't supported by real world gaming data.

My concern is that people are reading this thread and users like @lavoiejh and @eporrini pick this up and start having concerns, asking their players to roll differently and demand that the FGU developers drop everything and "fix" something that hasn't been proved is actually broken (in terms of affecting actual games), and even talking about moving to other VTTs as this is a differentiator (despite not being able to compare with other VTTs).

Based off simulated tests, in a purely test environment that doesn't accurately reflect a FG gaming session, you've discovered some interesting statistics that give you the basis for a hypothesis. The data you've presented so far is interesting, but it doesn't so far relate to a real gaming environment - which is what most of us playing FG are only interested in.

Unless you can prove that these statistics have a noticeable effect on people playing RPGs in FGU, then it's only a statistical analysis with no real world application.

Until you can actually reliably prove that there is a discernable impact in an actual game, please stop fanning the flames that can make people think FGU dice rolling truly is broken. It could cause them to use it as a differentiator with other VTTs and move on from FG (or not try FG in the first place) or will blame the fickle nature of dice rolling fate on FGU being broken, when it has not been proved that it is actually broken.

Weissrolf

December 4th, 2020, 12:20

No. This thread started with a graphic showing a bitmap image of 1 mio. consecutive D8 rolls. In this 1 mio. rolls there was a visible period every 200 rolls. No averaging, just rolls as bits = pixels.

After comparing FGU's "physical" D8 dice results with FGU's own "non-physical" /die command I concluded that FGU's physical dice engine seems to be bugged and does not create random numbers. FGU's non-physical /die pseudo-random number generator (PRNG) has its own flaws, but it is mostly functional compared to the current 3D implementation (judging by D8 rolls).

As a short-term hotfix I suggest to disable the 3D dice engine all together, as quickly as possible. As a long-term improvement I suggest to implement a better PRNG (like PCG) and give users the option to disable the 3D dice physics engine if it cannot be improved to match the /die engine.

https://i.imgur.com/0Yyhs5P.png https://i.imgur.com/BqXKT06.gif
My fault was to assume that FG's 3D rolls were meant to be random. Other persons' fault was to get hung up on the "as quickly as possible" part. Try reading my text in a somber and calm voice, it helps.

Jiminimonka

December 4th, 2020, 12:23

No. This thread started with a graphic showing a bitmap image of 1 mio. consecutive D8 rolls. In this 1 mio. rolls there was a visible period every 200 rolls. No averaging, just rolls as bits = pixels.

My fault was to even assume that FG's 3D rolls are meant to be random. Other persons' fault was to get hung up on the "as quickly as possible" part. Try reading my text in a somber and calm voice, it helps.

Stop making statements of fact without verifiable data. I think that's the just of the argument. Or get a scientist to verify your finding independently. Peer review.

I'm not sure if telling someone how to read your text is rude.

Trenloe

December 4th, 2020, 12:28

@Weissrolf - Move on to collecting and analysing real world data please. I really don't know how many times this has to be said.

The details I provide in post #125 above show that the statistics you're presenting don't translate into effecting actual FG gaming rolls. Until you can prove that there is a reproducible issue when using FG to game, not collect data in a test environment, then all of the nice cool graphs and statistical analysis really means nothing to FG gamers.

You have the basis for a hypothesis based off statistical analysis. Well done, nice work, that's cool. Now move on to prove this has a discernable effect to FG gamers in the real world.

Weissrolf

December 4th, 2020, 12:41

I proved that 3D click/automation rolls are not close to random and the data provided in your linked thread is unreliable. Doing the former was hard enough, doing the latter is harder even with reliable data. My suggestion was to fix the engine to better standard and/or offer an option for players to disable the faulty engine.

I deem the engine faulty, because it neither creates random rolls nor represents real physical dice. If SW's stance is that the dice engine has to achieve neither of these goals then maybe don't feature dice so prominently in your logo and video?!

https://i.imgur.com/V33VS1J.png

Anyway, our own groups will just be using the Instant Dice extension from now on and I suggest to point everyone to do the same should they be "feeling strange" about 3D dice results.

PS: Classic also introduces a 200 rolls period for D8 rolls, but the pattern is very different.

Weissrolf

December 4th, 2020, 13:02

My reply to a private message that should conclude this thread for me:

Problem with real campaign chat-logs is that I cannot compile these properly. Every line looks different and every roll from different chars use different boni.

We might be able to extract all D8 rolls in order to gather enough rolls for pattern recognition. We also don't know if those non D8 rolls in between change the pattern or not, which could be interesting to test, though.

All that being said: It is very obvious that neither SW nor forum/module/ruleset veterans are really interested in spending any energy on this. I kept providing so much analysis, data, graphs and explanations while the answers kept revolving around "You’re holding it wrong". There were few signs of "ok, let's take a look at this, albeit we are sure this isn't half as bad as it looks at first glance", instead we mostly got "you are making other users anxious".

So frankly, I have to tally this as a useful experience in data analysis, random number generators and social experiments. Our own groups have a workable alternative now, I suggest your groups to use the same until the system is improved.

Have a nice weekend and fun gaming!

Trenloe

December 4th, 2020, 13:05

I strongly disagree that the engine is faulty as far as effecting real world gaming in FG. You have yet to prove that. Maybe if people played 5,000 sessions and had somewhere near 1,000,000 rolls then the data presented would be relevant - but then it might not be, because FG initialises the dice rolling engine at the beginning of each session, or maybe running on a different day makes a difference, or maybe rolling different dice types in between rolls makes a difference, or maybe players connecting and disconnecting makes a difference, etc., etc..

Based off actual real world data, we don't see the same patterns that the statistics you present in this thread show. It has been reported that FG has on occasion rolled a 0 and reported it as such - so the campaign data presented is 100% real world data, and you can't discount it.

For gaming purposes, it is my view (based off real world data) that you can't use the simulated dice roll statistics presented in this thread as the basis for a claim of a faulty engine that impacts our gaming. Your next step is to try to prove that it effects our gaming in an actual gaming situation, if you can't then all of this is just interesting statistics with little relevance to our everyday FG gaming.

Weissrolf

December 4th, 2020, 13:10

Wait, FG rolls a 0 (zero) on a 3D physical D20 rolls? Yes, that absolutely increases my trust in the data and engine. Let's move on, there are no patterns to be seen here.

.|...|...|.
MMMMM
000000

Wow, just wow.

Trenloe

December 4th, 2020, 13:21

There were few signs of "ok, let's take a look at this, albeit we are sure this isn't half as bad as it looks at first glance".
I've spent a number of hours looking at real world data to see if the patterns you saw in testing were replicated. They weren't. So, yes - I took a look at it and I'm sure that this isn't as bad as it looks at first glance.

It is very obvious that neither SW nor forum/module/ruleset veterans are really interested in spending any energy on this.
Yes, you're probably right. Because as has been said many, many, many times - what you see doesn't repeat itself in actual play. Very few people are going to want to waste their energy banging their head trying to get that across.

... we mostly got "you are making other users anxious".
Because you *are* making people anxious - with nothing other than test statistics with no real world application to back them up. This is the Internet people will read what they want to read - e.g. FG competitors taking this away and using this against FG - where it (based off current data) has no discernable impact on a game, a person interested in FG seeing this as an active thread, having a read and thinking "Oh my! FG is terrible! I'll have a '7.5% lower chance to hit a 20' than I should!!!" - where there is currently no practical data to back this up as an in game issue.

Refer you to this anxious user's post as a classic example - based off your theoretical statistics:

I hope the Smiteworks team decides to take this seriously. Not having some level of integrity with rolling is a significant and foundational game issue. I am torn with using the instant dice extension as no rolling animation at all takes away from the joy of rolling on the virtual tabletop. I am leaning towards asking my players to roll manually whenever possible for now, even though I know certain number of rolls will get triggered through click or automatically.

I am re-petitioning the developers to work towards a real solution for this ASAP as a priority over other enhancements. I feel this is significant enough that I would use it as a differentiator when deciding on what virtual tabletop I would want to use to play on. The physics don't need to add up to "perfectly random", but we need something dependable enough that we don't feel like we are playing a rigged game with weighted dice.

My reply to a private message that should conclude this thread for me...
As this concludes it for you, I'm going to close this thread as it has more than ran it's course. If you find any in-gameplay issues in future please report them in a new thread.