Proof of concept/experimental, JSON, reduced "node" use

[celestian]

I've been trying to figure out the best way to improve load times for creatures placed into the Combat Tracker in the AD&D ruleset. It's kinda sluggish and I'd like to make it better. To that end I did some review of the code and the nodes that I might best "trim" out to reduce. As you can guess it's mostly Attacks, Powers and Inventory items.

My first experiment was to remove the inventory and "abilitynotes" nodes and all their children to npcs placed into the CT and replace those nested nodes with a single string of "node.getPath();node.getPath()" paths back to their source. The idea was I would use createControl for the view of those items showing them from the original source NPC.

It worked but the reduced feature set (can't edit really) really bothers me... but inventory and ability notes are not that big of a deal. Attacks are.

Attacks need a little more work to support. I started with the idea I did for inventory/ability notes and realized it wasn't really a good way to handle it. Someone had posted code that showed someone else using JSON strings in FG and it got me thinking, hell... I can store just about anything in a JSON string. I found a library that was opensource and began tinkering with it. After a bit I was able to get the weaponlist nested node and all the components stuffed into a single string entry on a npc in the CT and be able to "edit" the fields there.

Once I had it in JSON I began working on a UI window for it to display to tinker with it, see how it felt and test the load times.

The UI ended up being a intense learning experience and it game me a lot of ideas on how to to come work on the CT ui in general (I want to compact the npc views) but in the end it really didn't seem to make all that much difference in load times. I mean, it DOES make it load slightly faster but I think the added overhead of all the createControl() handling ate about 1/4 of the improvement or more.

I did notice some odd behavior in FG in general unrelated to these tweaks. I could close FG and start it and the first encounter I dropped into the CT was fairly spunky. If I cleared them and re-did it... each time it seemed to get slower (about 3-5th time it seemed to reach it's max slowness). I tested with my ruleset unmodified and modified and both seemed to experience this issue. Even tried the 5E one and seemed to be the same there. My guess is that it has something to do with memory management in the 32 bit app? Tho that's just speculation.

I'm going to keep tinkering with this a bit more to see what I can do but ... right not im mostly disappointed in the results. I don't think it's really worth the effect of losing the nodes (ease of manipulation) for the "slight" CT load times. Inventory items however I will probably continue to work with regardless so that people can use the Inventory tab and not dramatically affect CT load times. The benefits of inventory on ct npcs is worth it I think... it might end up being a "read only" mode. Meaning you'd need to have the npc's inventory setup the way you want before to drop them in the CT (which for me is 99.9% of the time anyway).

I say all that to solicit other's insight into this, perhaps someone has a method they use to help with these types of situations.

Here is the way I took the nodeNPC.weaponlist.* and converted it to a table then converted that to a JSON string.



-- pass a nodeNPC node and return the "weaponlist" children as a json string to be stored into a node as one single entry.
-- Once we have it in json style text we can use it like:
-- local aWeaponList = JSON.decode(DB.getValue(nodeCT,'weaponlist_json',"");
-- DB.setValue(nodeCT,'weaponlist_json',"string",JSON.encode(aWeaponList));
function getWeaponListAsJSONText(node)
local aWeaponsList = {};

for sID, nodeWeapon in pairs(DB.getChildren(node,"weaponlist")) do
local aWeapon = {};
aWeapon.sSource = nodeWeapon.getPath();
aWeapon.sID = sID;
aWeapon.sName = DB.getValue(nodeWeapon,"name","");
aWeapon.nAttackCurrent = DB.getValue(nodeWeapon,"attackview_weapon",0);
aWeapon.sAttackStat = DB.getValue(nodeWeapon,"attackstat","");
aWeapon.nAttackBonus = DB.getValue(nodeWeapon,"attackbonus",0);
aWeapon.nSpeedFactor = DB.getValue(nodeWeapon,"speedfactor",0);
aWeapon.nType = DB.getValue(nodeWeapon,"type",0);
aWeapon.nCarried = DB.getValue(nodeWeapon,"carried",0);
aWeapon.nMaxAmmo = DB.getValue(nodeWeapon,"maxammo",0);
aWeapon.nAmmo = DB.getValue(nodeWeapon,"ammo",0);
aWeapon.nLocked = DB.getValue(nodeWeapon,"locked",1);
-- aWeapon.sShortcutClass, aWeapon.sShortcutRecord = DB.getValue(nodeWeapon,"shortcut","","");
aWeapon.ItemNoteLocked = DB.getValue(nodeWeapon,"itemnote.locked",1);
aWeapon.ItemNoteName = DB.getValue(nodeWeapon,"itemnote.name","");
aWeapon.ItemNoteText = DB.getValue(nodeWeapon,"itemnote.text","");
aWeapon.aDamageList = {};
for sDMGid, nodeDamage in pairs(DB.getChildren(nodeWeapon,"damagelist")) do
local aDamage = {};
aDamage.sID = sDMGid;
aDamage.sDamageAsString = DB.getValue(nodeDamage,"damageasstring","");
aDamage.nBonus = DB.getValue(nodeDamage,"bonus",0);
aDamage.dDice = DB.getValue(nodeDamage,"dice","");
aDamage.sStat = DB.getValue(nodeDamage,"stat","");
aDamage.sType = DB.getValue(nodeDamage,"type","");
table.insert(aWeapon.aDamageList,aDamage);
end

-- sort damage by id so they appear as they do in weapons tab right now
local sort_byID = function( a,b ) return a.sID < b.sID end
table.sort(aWeapon.aDamageList, sort_byID);

-- add this weapon to weaponslist
table.insert(aWeaponsList,aWeapon);

-- Sort the weapons by name like they appear in list currently
local sort_byName = function( a,b ) return a.sName < b.sName end
table.sort(aWeaponsList, sort_byName);
end

local sJson = JSON.encode(aWeaponsList);

return sJson;
end




This is an example of how it turned out in the CT. I picked one of the most "nodey" npcs I had.

https://i.imgur.com/lWdbxQ8.png

The simple JSON library I used was this. I made some minor updates to it but it worked almost out of the gate.

https://gist.github.com/tylerneylon/59f4bcf316be525b30ab

[celestian]

Here is the bulk of the code I used to generate the subwindow contents of the weapons.



function onInit()
local nodeCT = getDatabaseNode();

DB.addHandler(DB.getPath(nodeCT, "weaponlist_json"), "onUpdate", buildWeaponsListForCTView);

buildWeaponsListForCTView();
end

function onClose()
DB.removeHandler(DB.getPath(nodeCT, "weaponlist_json"), "onUpdate", buildWeaponsListForCTView);
end

-- return the ID for this weapon entry
-- name_id-00001
function getMyID(sThisControl)
return sThisControl:match("_(id%-%d+)$");
end
-- return the ID for this damage entry
-- dmg_id-00004_id-00001
-- returns sDamageID, sWeaponID
function getMyDamageID(sThisControl)
return sThisControl:match("^dmg_(id%-%d+)_(id%-%d+)$");
end

-- populate rWeapon using the passed weapon id's record.
function getWeaponRecord(sWeaponID)
local rWeapon = {};
local nodeCT = getDatabaseNode();
local sWeaponsList = DB.getValue(nodeCT,"weaponlist_json","");
if sWeaponsList and sWeaponsList ~= '' then
-- decode the json style string weaponlist entry for this npc
local rWeaponList = JSON.decode(sWeaponsList);
for _,rWeaponSource in pairs(rWeaponList) do
if rWeaponSource.sID == sWeaponID then
rWeapon = rWeaponSource;
break;
end
end
end -- no weaponlist
return rWeapon;
end
-- get damage record from weapon
-- id-00001, id-00004
function getDamageRecord(sWeaponID,sDamageID)
local rDamage = {};
local rWeapon = getWeaponRecord(sWeaponID);
for _,rDMGSource in pairs(rWeapon.aDamageList) do
if rDMGSource.sID == sDamageID then
rDamage = rDMGSource;
break;
end
end

return rDamage;
end

-- set a new value to the JSON array stored on nodeCT, sWeaponID, sTag, sValue, sType
-- setWeaponRecordValue(nodeCT, id-00001,"nAmmo",3,"number")
function setWeaponRecordValue(nodeCT,sWeaponID,sTag,sValue, sType)
local sWeaponsList = DB.getValue(nodeCT,"weaponlist_json","");
if sWeaponsList and sWeaponsList ~= '' then
-- decode the json style string weaponlist entry for this npc
local aWeaponList = JSON.decode(sWeaponsList);
for nID,rWeaponSource in pairs(aWeaponList) do
if rWeaponSource.sID == sWeaponID then
if sType == "string" then
aWeaponList[nID][sTag] = tostring(sValue) or "";
elseif sType == "number" then
aWeaponList[nID][sTag] = tonumber(sValue) or 0;
else
aWeaponList[nID][sTag] = sValue;
end
local sWeaponListChanged = JSON.encode(aWeaponList);
DB.setValue(nodeCT,"weaponlist_json","string",sWeaponListChanged);
break;
end
end
end -- no weaponlist

end

-- this builds all the controls into the viewspace for weapons
function buildWeaponsListForCTView()
-- Debug.console("npc_weapons_ct.lua","buildWeaponsListForCTView","aControls[i].getName()",aControls[i].getName());
-- we remove existing controls and re-build from scratch
-- this is incase we eventually allow editing (add/delete items)
local aControls = getControls();
if #aControls > 0 then
for i=1 , #aControls do
local sControlName = aControls[i].getName();
if sControlName ~= 'contentanchor' then
aControls[i].destroy();
end
end
end

local nodeCT = getDatabaseNode();
local sWeaponsList = DB.getValue(nodeCT,"weaponlist_json","");
if sWeaponsList and sWeaponsList ~= '' then
-- decode the json style string weaponlist entry for this npc
local rWeaponList = JSON.decode(sWeaponsList);

local bRowShade = false;
for _,rWeapon in pairs(rWeaponList) do
local sFrame = nil;
if bRowShade then
sFrame = "rowshade";
end
--- VARS
local sControlInit = "init_" .. rWeapon.sID;
local sControlAttack = "attack_" .. rWeapon.sID;
local sControlName = "name_" .. rWeapon.sID;
local sControlType = "type_" .. rWeapon.sID;

--- CONTROLS
-- initiative roll
local controlInit = createControl("initiative_weapon_ct", sControlInit);
controlInit.setFrame(sFrame);
controlInit.setValue("[INIT:" .. rWeapon.nSpeedFactor .. "]");
controlInit.setReadOnly(true);
controlInit.setTooltipText("INITIATIVE");

-- attack name
local controlName = createControl("name_weapon_ct", sControlName);
controlName.setFrame(sFrame);
controlName.setValue(rWeapon.sName);
controlName.setReadOnly(true);
-- this sets height
controlName.setAnchor("top", sControlInit,"top","absolute",0);
-- this sets width
controlName.setAnchor("left", sControlInit,"right","absolute", 0);
controlName.setAnchor("right", 'contentanchor',"center","absolute",-100);

-- type of weapon
local controlType = createControl("type_weapon_ct", sControlType);
controlType.setFrame(sFrame);
controlType.setValue(rWeapon.nType);
controlType.setReadOnly(true);
controlType.setTooltipText("WEAPON TYPE");
-- this sets height
controlType.setAnchor("top", sControlName,"top","absolute",0);
controlType.setAnchor("bottom", sControlName,"bottom","absolute",0);
-- this sets width
controlType.setAnchor("left", sControlName,"right","absolute", 0);
controlType.setAnchor("right", sControlName,"right","absolute",20);

-- attack roll
local controlAttack = createControl("attack_weapon_ct", sControlAttack);
controlAttack.setFrame(sFrame);
-- make the attack label concise.
local sAttackString = "[ATK";
if rWeapon.nAttackCurrent ~= 0 then
sAttackString = sAttackString .. ":" .. rWeapon.nAttackCurrent .. "]";
else
sAttackString = sAttackString .. "]";
end
--
controlAttack.setValue(sAttackString);
controlAttack.setReadOnly(true);
controlAttack.setTooltipText("ATK");
-- this sets height
controlAttack.setAnchor("top", sControlType,"top","absolute",0);
-- this sets width
controlAttack.setAnchor("left", sControlType,"right","absolute", 0);
controlAttack.setAnchor("right", sControlType,"right","absolute",50);

-- add damage rolls
for nID, rDamage in pairs(rWeapon.aDamageList) do
local sControlDMG = "dmg_" .. rDamage.sID .. "_" .. rWeapon.sID;
local sDiceAsString = StringManager.convertDiceToString(rDamage.dDice, rDamage.nBonus);

-- take first letter of each type in the damage type string and uppercase it for compact view.
local aTypes = StringManager.split(rDamage.sType,",",true);
local sTypeLetters = "";
if #aTypes > 0 then
sTypeLetters = " ";
end
for nCount, sAType in pairs(aTypes) do
local sSep = ",";
if nCount >= #aTypes then
sSep = "";
end
sTypeLetters = sTypeLetters .. string.upper(sAType:sub(1,1)) .. sSep;
end
--

-- Damage control
local controlDMG = createControl("damage_weapon_ct", sControlDMG);
controlDMG.setFrame(sFrame);
local sDamageStringFinal = "[" .. sDiceAsString .. sTypeLetters .. "]";
local nSizeOfString = string.len(sDamageStringFinal);
controlDMG.setValue(sDamageStringFinal);
controlDMG.setReadOnly(true);
controlDMG.setTooltipText("Damage:" .. rDamage.sDamageAsString);
-- this sets height
controlDMG.setAnchor("top", sControlAttack,"top","absolute",0);
-- this sets width
controlDMG.setAnchor("left", sControlAttack,"right","relative", 0);
controlDMG.setAnchor("right", sControlAttack,"right","relative",(nSizeOfString*6));
end

-- add spacer to end so that it will shade entire line on frame
local controlSpacer = createControl("spacer_weapon_ct", 'spacer');
controlSpacer.setFrame(sFrame);
controlSpacer.setAnchor("top", sControlAttack,"top","absolute",0);
-- this sets width
controlSpacer.setAnchor("left", sControlAttack,"right","relative", 0);
controlSpacer.setAnchor("right", 'contentanchor',"right","absolute",0);

-- done ... NEXT!
bRowShade = not bRowShade;
end
end -- no sWeaponsList
end

[Bidmaron]

celestian, why not just write it out to xml file instead of json using the built-in import/export tools? Just curious why you thought using json was the way to go?

[celestian]

celestian, why not just write it out to xml file instead of json using the built-in import/export tools? Just curious why you thought using json was the way to go?

Data stored as an xml node is the cause of the "node" creation problem. You can replace 50+ nodes with a single node storing all the data as a json "string". The downside is you have to manage all your UI programmatically for those similarly stored objects.

JPG can explain it better than me but my understanding is the creation of nodes in FG takes significant time. (something to do with the backend processing of these objects with LUA). You don't notice it that much in rulesets like 5E because the system was built around avoiding extensive nodes. The 2E ruleset makes use of many and lots of them. The actions/weapons&powers sections specifically. Then adding those same sections to items in addition to inventory slots for npcs.

That was the impetus for this experimentation.

[kalmarjan]

How do I get up to speed on all this? I am both absolutely flabbergasted by this, and interested in learning more. (It helps that I essentially grew up with basic/2e, and matured playing 3.0/3.5. I can't tell you how many fights over rules I had. We could only dream of a program like this.)

[celestian]

How do I get up to speed on all this? I am both absolutely flabbergasted by this, and interested in learning more. (It helps that I essentially grew up with basic/2e, and matured playing 3.0/3.5. I can't tell you how many fights over rules I had. We could only dream of a program like this.)

Fortunately if you just want to play AD&D you can with the ruleset without having to know about these sorta things. All this is to try to do is come up with a better wheel I guess, or at least some axle grease? ;)

[Bidmaron]

Is this something that will get fixed with FGU, Celestian?

[celestian]

Is this something that will get fixed with FGU, Celestian?

Based on the replies from JPG, no. If might be less of an issue with more memory because of the 64 bit application (my experimentation here seemed to link some of this to memory use) but that is pure speculation on my part.

[Bidmaron]

Well, that’s a shame....

[Moon Wizard]

I have a hypothesis that removing database nodes as Lua objects would improve performance and load times a fair amount; but it's only an educated guess, and it breaks every single ruleset. Since every ruleset would be broken to make that change, it's a VERY large project to make the change, and fix every ruleset to work again with the revised API. At this point in time, it's not something that will be pursued, since that would be a large delay for FGU.

In the interim, celestian and I have been chatting back and forth on his findings and his UI re-engineering. I've been interested in revising the CT at some point, so his work is a way to see how people react to various UI changes.

Regards,
JPG

[LordEntrails]

Thanks Moon and Celestian. It's nice to see signs of collaborations and that new approaches are always being reviewed and considered.

[swbuza]

Fascinating discussion.