The tostring( ) lua function introduces "roundoff" errors!

[Minty23185Fresh]

In the following diatribe, I illustrate how a numeric value is corrupted by the lua tostring() function.

How do you like these equipment weights?
ItemWeights.png

No extensions are being used. I have the 5E Player's Handbook and Princes of the Apocalypse modules loaded (so that I have library data).

I have added the following custom filter to the 5E Items library (in rulesets\5E\scripts\data_library_5E.lua) at approximately line 116 (of Fantasy Grounds v3.3.3).

aRecords = {
   ...
   ["item"] = { 
      bExport = true,
      ...
      aPlayerListButtons = { "button_item_armor", "button_item_weapons" };
      aCustomFilters = {
         ["Type"] = { sField = "type" },
         ["Weight"] = { sField = "weight", sType = "number" },
     },
  },

Plus the following Debug.console() code at approximately line 455, in the getFilterValues() function of rulesets\CoreRPG\campaign\scripts\masterindex_wind ow.lua:

function getFilterValues(kCustomFilter, vNode)
   local vValues = {};
   local vCustomFilter = aCustomFilters[kCustomFilter];
   if vCustomFilter then
      if vCustomFilter.fGetValue then
         ...
      elseif vCustomFilter.sType == "boolean" then
         ...
      else
         local vValue = DB.getValue(vNode, vCustomFilter.sField);
         if kCustomFilter == "Weight" then
            Debug.console("masterindex_window.lua | getFilterValues() | DB   value=", vValue);    
         end
         if vCustomFilter.sType == "number" then
            vValues = { tostring(vValue or 0) };
            if kCustomFilter == "Weight" then
               Debug.console("masterindex_window.lua | getFilterValues() | Num value=", vValues);    
            end
         else
            local sValue;

Here is a small sample of the console:

Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #2
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'2'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #0.02
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'0.019999999552965'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #1
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'1'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #0.03
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'0.029999999329448'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #25
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'25'

Note the corruption of the data.

Most of this is for the Smiteworks powers to be. The important thing here for the casual observer is that an uncorrupted value is had from the database proven by the first Debug statement. The tostring function is executed and then the second Debug statement show the corrupted data.

I have ensured it is the tostring() function and not the recast to the table type that is doing the corrupting.

[Ikael]

tostring function does not corrupt the value, instead it shows the very true full value. After you read the value from DB you need to set the decimal precision manually and then tostring won't show too much data

For instance, with quick googling

function round(n, precision)
return math.floor(n*math.pow(10,precision)+0.5) / math.pow(10,precision)
end

[Minty23185Fresh]

tostring function does not corrupt the value, instead it shows the very true full value. After you read the value from DB you need to set the decimal precision manually and then tostring won't show too much data ..."

Okay. I buy in to that.
That implies that FG is doing a heck of a lot of these machinations behind the scenes. For instance that number, say 0.019999...., would have to be manipulated in such a manner when it is written to the db.xml file as 0.02, as well as, in this case, when it's written to the console, as 0.02.

[pindercarl]

Okay. I buy in to that.
That implies that FG is doing a heck of a lot of these machinations behind the scenes. For instance that number, say 0.019999...., would have to be manipulated in such a manner when it is written to the db.xml file as 0.02, as well as, in this case, when it's written to the console, as 0.02.

What you're looking at is floating point imprecision. It is not unique to Fantasy Grounds or LUA. It's nature of floating points (non-integer) values on computer hardware. LUA uses double-precision floating point values (64-bit vs 32-bit). The imprecision will be less, but still occurs. Without looking at the XML serialization, rounding to a specific number of significant digits is common when writing out floating point values in any language. In C++, for example, printf_s( "%.2f", fp ); with write out the value of fp to two decimal places. 0.02999 with be written as 0.3. There may be a similar LUA formatting function available in Fantasy Grounds. I'm not sure. I think the LUA is, string.format('%f3.2', vValue) Where 3 is the amount of numbers before the decimal point and 2 is amount of numbers after.

[Minty23185Fresh]

What you're looking at is floating point imprecision. It is not unique to Fantasy Grounds or LUA. It's nature of floating points (non-integer) values on computer hardware....

There may be a similar LUA formatting function available in Fantasy Grounds. I'm not sure. I think the LUA is, string.format('%f3.2', vValue)....

Thank you for revisiting floating point imprecision with me. I am familiar with it. I've been writing code for a while now, including floating point math at the machine language level for microcomputers in the 1970's.

I got caught with my pants down, with a simple recast from number to string in a language, lua, that I am relatively new to. I do like the string.format solution, thanks for suggesting it. Having to add the additional levels of complexity, when all I want to do it is populate a computer box, is well, exasperating.

[LordEntrails]

...
I got caught with my pants down....

Good thing this isn't Instagram. That might have been scary!!

[Minty23185Fresh]

Good thing this isn't Instagram. That might have been scary!!

On so many levels!

[Minty23185Fresh]

A follow up question of Ikael and pindercarl, and of course anyone else who might want to jump in, if I may?

If the true value of the number is 0.003000000026077, why doesn't Debug.console( ) show me that value? Debug displayed 0.003, the same number as the 0.003 in the db.xml file. Does the code in the Debug library just want to be friendly and not hurt my brain with all those zeros (or nines, as the case may be)? It wasn't until I recast the number to a string that I learned the actual numeric value. Seems to me I remember something like that in one of Microsoft's IDEs, maybe Visual Basic? C++ 6.0?

Don't get me wrong, I completely on board with your explanations, they're spot on, I'm just surprised, a little sheepish and feeling downright foolish. Had Debug.console() given me the actual value, I'd have known what the issue was and we wouldn't be slogging around in this thread.

Should I practice defensive debugging in addition to defensive coding and if I have a number that might have a fractional part I should send it to Debug like this Debug.console("Actual value: ", tostring(num)); ? Seems a bit silly.

[Moon Wizard]

I have a function in the FG client code that perform floating point “rounding” for display purposes. I believe it is set the round at two decimals of precision, but I’d have to review the code to be positive.

Regards,
JPG

[Minty23185Fresh]

A little earlier I tried this, just for fun:

Debug.console(num, tostring(num), tonumber(tostring(num)));

The results were:

0.03, 0.03000000026077, 0.03

To tell you the truth, I'm not sure what I expected. But I'm not surprised by the results.
It does reinforce my suspicion that Debug.console() does massage the data, probably performing some rounding, before outputting the strings to the console. Something like, if the absolute value between the number and x^-nth is less than say 1e-10 then output x^-nth, as opposed to the num. (I hope I have that right... For this example if 0.03000000026077 - 0.03 is less than 1e-10 then output 0.03.)

I tried to test the theory with:
for i=0.1,0.9,0.1 do
local x = i;
for j=1,10,1 do
Debug.console(x, tostring(x), tonumber(tostring(x)));
x = x / 10;
end
end

But didn't really get anywhere.

Thanks, Moon Wizard. I'd like to know the secret of Debug.console(). As far as I recall, back from the C days most format(%w.pf, ..) results in pre or post padding with either 0's or spaces. Something that isn't occurring with Debug.console() output.