The tostring( ) lua function introduces "roundoff" errors!

[Minty23185Fresh]

In the following diatribe, I illustrate how a numeric value is corrupted by the lua tostring() function.

How do you like these equipment weights?
21420

No extensions are being used. I have the 5E Player's Handbook and Princes of the Apocalypse modules loaded (so that I have library data).

I have added the following custom filter to the 5E Items library (in rulesets\5E\scripts\data_library_5E.lua) at approximately line 116 (of Fantasy Grounds v3.3.3).


aRecords = {
...
["item"] = {
bExport = true,
...
aPlayerListButtons = { "button_item_armor", "button_item_weapons" };
aCustomFilters = {
["Type"] = { sField = "type" },
["Weight"] = { sField = "weight", sType = "number" },
},
},


Plus the following Debug.console() code at approximately line 455, in the getFilterValues() function of rulesets\CoreRPG\campaign\scripts\masterindex_wind ow.lua:


function getFilterValues(kCustomFilter, vNode)
local vValues = {};
local vCustomFilter = aCustomFilters[kCustomFilter];
if vCustomFilter then
if vCustomFilter.fGetValue then
...
elseif vCustomFilter.sType == "boolean" then
...
else
local vValue = DB.getValue(vNode, vCustomFilter.sField);
if kCustomFilter == "Weight" then
Debug.console("masterindex_window.lua | getFilterValues() | DB value=", vValue);
end
if vCustomFilter.sType == "number" then
vValues = { tostring(vValue or 0) };
if kCustomFilter == "Weight" then
Debug.console("masterindex_window.lua | getFilterValues() | Num value=", vValues);
end
else
local sValue;


Here is a small sample of the console:


Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB value=' | #2
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'2'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB value=' | #0.02
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'0.019999999552965'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB value=' | #1
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'1'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB value=' | #0.03
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'0.029999999329448'
Runtime Notice: s'masterindex_window.lua | getFilterValues() | DB value=' | #25
Runtime Notice: s'masterindex_window.lua | getFilterValues() | Num value=' | s'25'


Note the corruption of the data.

Most of this is for the Smiteworks powers to be. The important thing here for the casual observer is that an uncorrupted value is had from the database proven by the first Debug statement. The tostring function is executed and then the second Debug statement show the corrupted data.

I have ensured it is the tostring() function and not the recast to the table type that is doing the corrupting.

[Ikael]

tostring function does not corrupt the value, instead it shows the very true full value. After you read the value from DB you need to set the decimal precision manually and then tostring won't show too much data

For instance, with quick googling

function round(n, precision)
return math.floor(n*math.pow(10,precision)+0.5) / math.pow(10,precision)
end

[Minty23185Fresh]

tostring function does not corrupt the value, instead it shows the very true full value. After you read the value from DB you need to set the decimal precision manually and then tostring won't show too much data ..."

Okay. I buy in to that.
That implies that FG is doing a heck of a lot of these machinations behind the scenes. For instance that number, say 0.019999...., would have to be manipulated in such a manner when it is written to the db.xml file as 0.02, as well as, in this case, when it's written to the console, as 0.02.

[pindercarl]

Okay. I buy in to that.
That implies that FG is doing a heck of a lot of these machinations behind the scenes. For instance that number, say 0.019999...., would have to be manipulated in such a manner when it is written to the db.xml file as 0.02, as well as, in this case, when it's written to the console, as 0.02.

What you're looking at is floating point imprecision. It is not unique to Fantasy Grounds or LUA. It's nature of floating points (non-integer) values on computer hardware. LUA uses double-precision floating point values (64-bit vs 32-bit). The imprecision will be less, but still occurs. Without looking at the XML serialization, rounding to a specific number of significant digits is common when writing out floating point values in any language. In C++, for example, printf_s( "%.2f", fp ); with write out the value of fp to two decimal places. 0.02999 with be written as 0.3. There may be a similar LUA formatting function available in Fantasy Grounds. I'm not sure. I think the LUA is, string.format('%f3.2', vValue) Where 3 is the amount of numbers before the decimal point and 2 is amount of numbers after.

[Minty23185Fresh]

What you're looking at is floating point imprecision. It is not unique to Fantasy Grounds or LUA. It's nature of floating points (non-integer) values on computer hardware....

There may be a similar LUA formatting function available in Fantasy Grounds. I'm not sure. I think the LUA is, string.format('%f3.2', vValue)....

Thank you for revisiting floating point imprecision with me. I am familiar with it. I've been writing code for a while now, including floating point math at the machine language level for microcomputers in the 1970's.

I got caught with my pants down, with a simple recast from number to string in a language, lua, that I am relatively new to. I do like the string.format solution, thanks for suggesting it. Having to add the additional levels of complexity, when all I want to do it is populate a computer box, is well, exasperating.

[LordEntrails]

...
I got caught with my pants down....
Good thing this isn't Instagram. That might have been scary!!

[Minty23185Fresh]

Good thing this isn't Instagram. That might have been scary!!
On so many levels!

[Minty23185Fresh]

A follow up question of Ikael and pindercarl, and of course anyone else who might want to jump in, if I may?

If the true value of the number is 0.003000000026077, why doesn't Debug.console( ) show me that value? Debug displayed 0.003, the same number as the 0.003 in the db.xml file. Does the code in the Debug library just want to be friendly and not hurt my brain with all those zeros (or nines, as the case may be)? It wasn't until I recast the number to a string that I learned the actual numeric value. Seems to me I remember something like that in one of Microsoft's IDEs, maybe Visual Basic? C++ 6.0?

Don't get me wrong, I completely on board with your explanations, they're spot on, I'm just surprised, a little sheepish and feeling downright foolish. Had Debug.console() given me the actual value, I'd have known what the issue was and we wouldn't be slogging around in this thread.

Should I practice defensive debugging in addition to defensive coding and if I have a number that might have a fractional part I should send it to Debug like this Debug.console("Actual value: ", tostring(num)); ? Seems a bit silly.

[Moon Wizard]

I have a function in the FG client code that perform floating point “rounding” for display purposes. I believe it is set the round at two decimals of precision, but I’d have to review the code to be positive.

Regards,
JPG

[Minty23185Fresh]

A little earlier I tried this, just for fun:

Debug.console(num, tostring(num), tonumber(tostring(num)));

The results were:

0.03, 0.03000000026077, 0.03

To tell you the truth, I'm not sure what I expected. But I'm not surprised by the results.
It does reinforce my suspicion that Debug.console() does massage the data, probably performing some rounding, before outputting the strings to the console. Something like, if the absolute value between the number and x^-nth is less than say 1e-10 then output x^-nth, as opposed to the num. (I hope I have that right... For this example if 0.03000000026077 - 0.03 is less than 1e-10 then output 0.03.)

I tried to test the theory with:
for i=0.1,0.9,0.1 do
local x = i;
for j=1,10,1 do
Debug.console(x, tostring(x), tonumber(tostring(x)));
x = x / 10;
end
end

But didn't really get anywhere.

Thanks, Moon Wizard. I'd like to know the secret of Debug.console(). As far as I recall, back from the C days most format(%w.pf, ..) results in pre or post padding with either 0's or spaces. Something that isn't occurring with Debug.console() output.

[Andraax]

Moon said he has code that rounds numbers displayed to 2 digits. The results of your display were a number, a string, and a number. The two numbers were rounded to 2 digits, the string was not, because it is not a number.

[Andraax]

perl -E '@n=(3,0.03000000026077,3.2); say sprintf "%.2g", $_ foreach @n;'
3
0.03
3.2

[Minty23185Fresh]

perl ......


Thanks for the perl, Andraax. :confused:

[Andraax]

Thanks for the perl, Andraax. :confused:

Perl uses C / C++ printf formats, as does LUA. Just demonstrating that some formats don't pad with 0s or spaces, as you said in your post.


Thanks, Moon Wizard. I'd like to know the secret of Debug.console(). As far as I recall, back from the C days most format(%w.pf, ..) results in pre or post padding with either 0's or spaces. Something that isn't occurring with Debug.console() output.

[Minty23185Fresh]

Working on my own routine to do the rounding I need to my desired number of places...

One thing that I have noticed, every time I Debug.console() a number, if there are a large number of decimal places, Debug performs rounding before display. It appears to top out at 4 decimal places. It is as if there is an environment setting for it (set by Smiteworks? or internal to lua itself?).

Examples:
n = 15.1999999999999999; Debug.console(n); --> 15.2
n = 26.2300000000009879; Debug.console(n); --> 26.23
n = 37.7349999990005329; Debug.console(n); --> 37.735
n = 48.1111000000000812; Debug.console(n); --> 48.1111
n = 51.5432799999999210; Debug.console(n); --> 51.5433 not 51.54328 !!

[Minty23185Fresh]

Perl uses C / C++ printf formats, as does LUA. Just demonstrating that some formats don't pad with 0s or spaces, as you said in your post.

Ah. Thank you. Is that what the "g" is in the format "%.2g", it prevents trailing 0's ?

[Andraax]

Yes.

Try:

Debug.console(string.format("%.5g",n))

[Minty23185Fresh]

Thanks Andraax, I will.
I've just about finished the function I'm writing, but that little pearl might be exactly what I need.

[Minty23185Fresh]

@Andraax:
Not quite it. It prints 5 digits, including the integer part. So 123.456781. ---> 123.45. I want 123.45678. Five fractional digits regardless of the magnitude of the integer part.

I tried adding a width, as in %10.5g, but that pre-pads with spaces and still prints a total of 5 digits.
I remember this with C/C++. How a language as powerful as these could be so aggravating in this respect!

I don't need to waste your time as well as my own with such a trivial problem. Do you have an Internet resource for regular expressions that you like? The Lua manual is lacking. Other sites I have visited in the last hour have a nibble here, a byte there, and maybe a couple words. But most are incomplete.

Thanks in advance.

[Bidmaron]

There are no regular expressions built into lua for FG. If you are talking about Lua search expressions, I have looked far and wide for a lua search expression tester but it just doesn’t exist.
As for formatting, the lua 5.1 manual says the formatting instructions are identical to C language print formatting. Away from my computer but you should be able to find plenty on C print formatting.
The string manager in FG has a trim function you can use to strip off those padding blanks (StringManager.trim)

[Andraax]

Just tested this:


perl -E '@n=(3,0.03000000026077,3.2,123.456781); say sprintf "%.5g", $_ foreach @n;'
3
0.03
3.2
123.46

Fixed it like so:


perl -E '@n=(3,0.03000000026077,3.2,123.456781); say sprintf "%.*g", log($_)/log(10)+5+1, $_ foreach @n;'
3
0.03
3.2
123.45678

Equivalent code in FG would be something like:


Debug.console(string.format("%.*g",math.log10(n)+5+1,n))

[Andraax]

And https://www.regexplanet.com/ has a good set of RE testers that I use for Perl, Python, Ruby, etc. But it doesn't have one for LUA.

[damned]

Minty why are you doing this? Why are you not just using the numbers as stored in the DB?
Forgive my stupid question...

[Minty23185Fresh]

Minty why are you doing this? Why are you not just using the numbers as stored in the DB?

Mid way through this, I was wondering why, myself! :hurt:

Please take a look at the left panel in the screenshot below. When the numbers (fractional weights) in the database are recast as strings to populate the combobox, their values are displayed in all their wonder. Me being as compulsive as I am, recognized them as 0.003, 0.025, 0.... and so I wanted them to display correctly.

I thought it was a bug in FG, but as Ikael, pindercarl and others have pointed out, it is the issue with precision of base 10 numbers stored in a base 2 environment. It doesn't really present itself until the numeric is recast as a string.

I just finished my routine about 15 minutes ago, see the right panel in the screen shot. I used a little of this and a little of that from most everyone including Ikael and Andraax.

Thanks to all for your help.

21437

[damned]

Ok... so why convert them to strings when they are numbers?
If the conversion to string is causing the challenge dont convert to string.
Do you need to search weights of items?
Do you need to search costs?
Are you converting numbers to strings so that you can compare string search results with numbers? Is there a better way? Have number search fields for numbers?
Why do you even need to search numbers?

[Minty23185Fresh]

@Damed....
Most if not all those questions can be addressed with a couple answers.

Did I need to support all those fields? I honestly doubt it. Certainly not for me, but I am trying to avail all fields to filtering for anybody at any time, just in case they might want to. Sure as heck as I didn't, some one in the future would ask me to add support for it. A good example where number filtering is relevant is CR and or XP filtering. As a DM building a campaign having the ability to look up either via a filter seems very useful to me. Weights? Costs? Maybe, probably not. But they're available now.

Why convert numbers to strings. The infrastructure in the core ruleset populates drop down combo boxes with strings. If I used the provided infrastructure to effect this extension, I didn't have to do a lot of new coding. I didn't want to reinvent or redesign everything. Being miminally invasive seems a wiser way to go. To work within those limitations required me to recast data to string types. That also required my to engineer some special sort routines to handle sorting of numbers cast as strings (e.g. 10 comes before 2 as a string but after 2 as a number). By far the worst stumbling block was that that spawned this thread: recasting the fractional numbers to strings, and I think weights were the only numbers with fractional parts.

[Minty23185Fresh]

Oh yeah by the way one of the custom sort routines I implemented was sorting hit dice. Sorting 3d6 versus 10d4 versus 7d20. It was one of the more interesting. Do I sort first by die (d10 before d20?) or by the number of dice? I chose to sort by die first, then count then bonus. So 10d4 comes before 1d6 and 3d10 comes before 3d10+6.

[Bidmaron]

Oh yeah by the way one of the custom sort routines I implemented was sorting hit dice. Sorting 3d6 versus 10d4 versus 7d20. It was one of the more interesting. Do I sort first by die (d10 before d20?) or by the number of dice? I chose to sort by die first, then count then bonus. So 10d4 comes before 1d6 and 3d10 comes before 3d10+6.

Never thought about that, Minty. If it makes a difference, I agree with how you decided to do it. The less-preferred alternatives (my opinion) would be max theoretical HP or average HP from the die.

I also laud your efforts to support all fields. This is one of those things that you probably rarely do to search on something like item weight, but maybe you are 3 pounds from your next encumbrance-class rollover, and you want to see what you could cram in your backpack.

[Minty23185Fresh]

Thanks Bidmaron. I appreciate that.

[Bidmaron]

Us old farts have to stick together (57 and counting).