STAR TREK 2d20
Page 1 of 3 123 Last
  1. #1
    Minty23185Fresh's Avatar
    Join Date
    Dec 2015
    Location
    Goldstone, CA, USA
    Posts
    1,211
    Blog Entries
    29

    The tostring( ) lua function introduces "roundoff" errors!

    In the following diatribe, I illustrate how a numeric value is corrupted by the lua tostring() function.

    How do you like these equipment weights?
    ItemWeights.png

    No extensions are being used. I have the 5E Player's Handbook and Princes of the Apocalypse modules loaded (so that I have library data).

    I have added the following custom filter to the 5E Items library (in rulesets\5E\scripts\data_library_5E.lua) at approximately line 116 (of Fantasy Grounds v3.3.3).
    Code:
    aRecords = {
       ...
       ["item"] = { 
          bExport = true,
          ...
          aPlayerListButtons = { "button_item_armor", "button_item_weapons" };
          aCustomFilters = {
             ["Type"] = { sField = "type" },
             ["Weight"] = { sField = "weight", sType = "number" },
         },
      },
    Plus the following Debug.console() code at approximately line 455, in the getFilterValues() function of rulesets\CoreRPG\campaign\scripts\masterindex_wind ow.lua:
    Code:
    function getFilterValues(kCustomFilter, vNode)
       local vValues = {};
       local vCustomFilter = aCustomFilters[kCustomFilter];
       if vCustomFilter then
          if vCustomFilter.fGetValue then
             ...
          elseif vCustomFilter.sType == "boolean" then
             ...
          else
             local vValue = DB.getValue(vNode, vCustomFilter.sField);
             if kCustomFilter == "Weight" then
                Debug.console("masterindex_window.lua | getFilterValues() | DB   value=", vValue);    
             end
             if vCustomFilter.sType == "number" then
                vValues = { tostring(vValue or 0) };
                if kCustomFilter == "Weight" then
                   Debug.console("masterindex_window.lua | getFilterValues() | Num value=", vValues);    
                end
             else
                local sValue;
    Here is a small sample of the console:
    Code:
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #2
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'2'
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #0.02
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'0.019999999552965'
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #1
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'1'
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #0.03
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'0.029999999329448'
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB   value=' | #25
    Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'25'
    Note the corruption of the data.

    Most of this is for the Smiteworks powers to be. The important thing here for the casual observer is that an uncorrupted value is had from the database proven by the first Debug statement. The tostring function is executed and then the second Debug statement show the corrupted data.

    I have ensured it is the tostring() function and not the recast to the table type that is doing the corrupting.
    Last edited by Minty23185Fresh; November 10th, 2017 at 21:19.
    Current Projects:
    Always...
    Community Contributions:
    Extensions: Bardic Inspiration, Druid Wild Shapes, Local Dice Tower, Library Field Filters
    Tutorial Blog Series: "A Neophyte Tackles (coding) the FG Extension".

  2. #2
    Ikael's Avatar
    Join Date
    Jan 2008
    Location
    Finland/Joensuu
    Posts
    2,384
    tostring function does not corrupt the value, instead it shows the very true full value. After you read the value from DB you need to set the decimal precision manually and then tostring won't show too much data

    For instance, with quick googling

    function round(n, precision)
    return math.floor(n*math.pow(10,precision)+0.5) / math.pow(10,precision)
    end
    Last edited by Ikael; November 10th, 2017 at 22:39.
    "Alright, you primitive screwheads, listen up: THIS... is my BOOMSTICK!" -- Ash Williams, Army of Darkness

    Post your SavageWorlds ruleset feature requests and issue reports here!

  3. #3
    Minty23185Fresh's Avatar
    Join Date
    Dec 2015
    Location
    Goldstone, CA, USA
    Posts
    1,211
    Blog Entries
    29
    Quote Originally Posted by Ikael View Post
    tostring function does not corrupt the value, instead it shows the very true full value. After you read the value from DB you need to set the decimal precision manually and then tostring won't show too much data ..."
    Okay. I buy in to that.
    That implies that FG is doing a heck of a lot of these machinations behind the scenes. For instance that number, say 0.019999...., would have to be manipulated in such a manner when it is written to the db.xml file as 0.02, as well as, in this case, when it's written to the console, as 0.02.

  4. #4
    pindercarl's Avatar
    Join Date
    Jan 2015
    Posts
    988
    Blog Entries
    2
    Quote Originally Posted by Minty23185Fresh View Post
    Okay. I buy in to that.
    That implies that FG is doing a heck of a lot of these machinations behind the scenes. For instance that number, say 0.019999...., would have to be manipulated in such a manner when it is written to the db.xml file as 0.02, as well as, in this case, when it's written to the console, as 0.02.
    What you're looking at is floating point imprecision. It is not unique to Fantasy Grounds or LUA. It's nature of floating points (non-integer) values on computer hardware. LUA uses double-precision floating point values (64-bit vs 32-bit). The imprecision will be less, but still occurs. Without looking at the XML serialization, rounding to a specific number of significant digits is common when writing out floating point values in any language. In C++, for example, printf_s( "%.2f", fp ); with write out the value of fp to two decimal places. 0.02999 with be written as 0.3. There may be a similar LUA formatting function available in Fantasy Grounds. I'm not sure. I think the LUA is, string.format('%f3.2', vValue) Where 3 is the amount of numbers before the decimal point and 2 is amount of numbers after.

  5. #5
    Minty23185Fresh's Avatar
    Join Date
    Dec 2015
    Location
    Goldstone, CA, USA
    Posts
    1,211
    Blog Entries
    29
    Quote Originally Posted by pindercarl View Post
    What you're looking at is floating point imprecision. It is not unique to Fantasy Grounds or LUA. It's nature of floating points (non-integer) values on computer hardware....

    There may be a similar LUA formatting function available in Fantasy Grounds. I'm not sure. I think the LUA is, string.format('%f3.2', vValue)....
    Thank you for revisiting floating point imprecision with me. I am familiar with it. I've been writing code for a while now, including floating point math at the machine language level for microcomputers in the 1970's.

    I got caught with my pants down, with a simple recast from number to string in a language, lua, that I am relatively new to. I do like the string.format solution, thanks for suggesting it. Having to add the additional levels of complexity, when all I want to do it is populate a computer box, is well, exasperating.
    Last edited by Minty23185Fresh; November 11th, 2017 at 01:21.

  6. #6
    LordEntrails's Avatar
    Join Date
    May 2015
    Location
    -7 UTC
    Posts
    17,374
    Blog Entries
    9
    Quote Originally Posted by Minty23185Fresh View Post
    ...
    I got caught with my pants down....
    Good thing this isn't Instagram. That might have been scary!!

    Problems? See; How to Report Issues, Bugs & Problems
    On Licensing & Distributing Community Content
    Community Contributions: Gemstones, 5E Quick Ref Decal, Adventure Module Creation, Dungeon Trinkets, Balance Disturbed, Dungeon Room Descriptions
    Note, I am not a SmiteWorks employee or representative, I'm just a user like you.

  7. #7
    Minty23185Fresh's Avatar
    Join Date
    Dec 2015
    Location
    Goldstone, CA, USA
    Posts
    1,211
    Blog Entries
    29
    Quote Originally Posted by LordEntrails View Post
    Good thing this isn't Instagram. That might have been scary!!
    On so many levels!
    Current Projects:
    Always...
    Community Contributions:
    Extensions: Bardic Inspiration, Druid Wild Shapes, Local Dice Tower, Library Field Filters
    Tutorial Blog Series: "A Neophyte Tackles (coding) the FG Extension".

  8. #8
    Minty23185Fresh's Avatar
    Join Date
    Dec 2015
    Location
    Goldstone, CA, USA
    Posts
    1,211
    Blog Entries
    29
    A follow up question of Ikael and pindercarl, and of course anyone else who might want to jump in, if I may?

    If the true value of the number is 0.003000000026077, why doesn't Debug.console( ) show me that value? Debug displayed 0.003, the same number as the 0.003 in the db.xml file. Does the code in the Debug library just want to be friendly and not hurt my brain with all those zeros (or nines, as the case may be)? It wasn't until I recast the number to a string that I learned the actual numeric value. Seems to me I remember something like that in one of Microsoft's IDEs, maybe Visual Basic? C++ 6.0?

    Don't get me wrong, I completely on board with your explanations, they're spot on, I'm just surprised, a little sheepish and feeling downright foolish. Had Debug.console() given me the actual value, I'd have known what the issue was and we wouldn't be slogging around in this thread.

    Should I practice defensive debugging in addition to defensive coding and if I have a number that might have a fractional part I should send it to Debug like this Debug.console("Actual value: ", tostring(num)); ? Seems a bit silly.

  9. #9
    I have a function in the FG client code that perform floating point “rounding” for display purposes. I believe it is set the round at two decimals of precision, but I’d have to review the code to be positive.

    Regards,
    JPG

  10. #10
    Minty23185Fresh's Avatar
    Join Date
    Dec 2015
    Location
    Goldstone, CA, USA
    Posts
    1,211
    Blog Entries
    29
    A little earlier I tried this, just for fun:

    Debug.console(num, tostring(num), tonumber(tostring(num)));

    The results were:

    0.03, 0.03000000026077, 0.03

    To tell you the truth, I'm not sure what I expected. But I'm not surprised by the results.
    It does reinforce my suspicion that Debug.console() does massage the data, probably performing some rounding, before outputting the strings to the console. Something like, if the absolute value between the number and x^-nth is less than say 1e-10 then output x^-nth, as opposed to the num. (I hope I have that right... For this example if 0.03000000026077 - 0.03 is less than 1e-10 then output 0.03.)

    I tried to test the theory with:
    for i=0.1,0.9,0.1 do
    local x = i;
    for j=1,10,1 do
    Debug.console(x, tostring(x), tonumber(tostring(x)));
    x = x / 10;
    end
    end

    But didn't really get anywhere.

    Thanks, Moon Wizard. I'd like to know the secret of Debug.console(). As far as I recall, back from the C days most format(%w.pf, ..) results in pre or post padding with either 0's or spaces. Something that isn't occurring with Debug.console() output.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Starfinder Playlist

Log in

Log in