Floating Point Precision differences

[bmos]

I'm just about done a big update to one of my extensions and went back to test it in Classic and one of the features is totally nonfunctional (but works beautifully in Unity).
Has the floating point precision been upgraded in Unity?

I did try to do a search for this, but mostly found old threads on the 'issue' in FGC.

[Moon Wizard]

Not that I'm aware of. Both FGC and FGU use "double" data types for numbers. However, they do use different implementations of almost every internal function (C# vs. C++). I'd need more information on what APIs are reporting differently.

Regards,
JPG

[bmos]

Here is a simple function to demonstrate:


local function onTest()
local nBigNumber = 12345678910
Debug.chat('nBigNumber: ' .. nBigNumber)
Debug.chat(nBigNumber)
end

In Classic:
39675

In Unity:
39676

This might actually just be something different about how debug.chat works, but it's really been making my head hurt!
Based on this testing, it looks like the actual value is still there (since it shows when concatenated into a string).
I will use the approach of storing the number in a string for now so I can see what is actually happening in my extension's code.

[bmos]

Zygmunt Molotch might have just given me the answer.
Apparently lua has 64-bit libraries that allow extra precision.
Perhaps you have this in Unity and not in Classic?

[Moon Wizard]

Not sure. I believe they use the same libraries; but the interface to the libraries is 32-bit on FGC and 64-bit on FGU. That may or may not be the issue. "double" is a 64-bit data type in both C++ and C#; and shouldn't vary between 32-bit vs. 64-bit libraries.

The Debug output may just be different; since the code is very different between FGC/FGU.

The real question is where the number differences are showing up in the code that is causing issues. (number -> string conversion, API return value precision, etc.)

Regards,
JPG

[bmos]

The real question is where the number differences are showing up in the code that is causing issues. (number -> string conversion, API return value precision, etc.)Yeah, it looks like the problem I thought was the problem was just inaccurate debug output. I changed my debug.chat(nVar) usage to debug.chat("" .. nVar) so I can get usable info.
Thanks for your time JPG, it is always appreciated :)

[bmos]

Alright, I found where the two are behaving differently.



DB.setValue(window.getDatabaseNode(), 'starttime', 'number', nPreciseTime)
Debug.chat('set: ' .. nPreciseTime, 'check set: ' .. DB.getValue(window.getDatabaseNode(), 'starttime', 0))


These lines are directly after each other and result in the attached output.

I have no handlers associated with that node. The code I included is in the script tag on a buttoncontrol object (no extra templates). I tried changing 'starttime' to another name that I have not used before to verify that nothing in my code was targeting that node and it behaved the same way.

In Unity, these numbers match.

[Moon Wizard]

Ah, found it. FGC stores numbers has single floats internally; and FGU was upgraded to use double floats internally.

Regards,
JPG

[Moon Wizard]

You might need to divide before you can save the number to write code you can use in both.

Regards,
JPG

[bmos]

You might need to divide before you can save the number to write code you can use in both.

Regards,
JPGOk! Thank you so much.

EDIT: For anyone who ends up here with the same issue in the future, here is an even better workaround (in some ways):

Save it as a string in the database with DB.setValue(parentnode, 'nodename', 'string', tostring(nVar))
Then, on the other end, use tonumber(DB.getValue(parentnode, 'nodename'))

There is still a potential for some inaccuracy (according to a friend of mine, at least), but not nearly as much as single-precision floating point.