The cost of DB.setCategory() seems extreme?

[celestian]

So I was adding a feature to Author to bulk create stub story chapter(groups) with story entries in each. The code to process it is very simple.



for nChapter = nChapterStart, nChapterStart+nChapterCount-1 do
local sChapterString = string.format("%02d",nChapter);
for nStory = nStoryStart, nStoryStart+nStoryCount-1 do
local nodeStory = DB.createChild('encounter');
local sStoryString = string.format("%03d",nStory);
Debug.console("story_bulk_stubs.lua","processBulkCreations","Chapter.Story: ",sChapterString .. "." .. sStoryString);
DB.setValue(nodeStory,"name","string",sChapterString .. "." .. sStoryString);
DB.setCategory(nodeStory, sChapterString);
end
end


This creates "XX.XXX" story and places it into a category "XX".

That is all it does, but if you watch it create 10 story entries with 3 chapters it's ponderously slow. If you try 10 chapters with 100 entries each it'll crash.

Is there some nuance about this im missing?

[Varsuuk]

Hey Celestian, I haven't written Reference pages, I have only loaded Author one time to check it out cos it looked cool.
That and my inexperience with LUA/FG coding means I likely have little to add but my thoughts:

If you commented out the last line: "DB.setCategory(nodeStory, sChapterString);" --> what kind of increase in time are we talking? Also, does it crash if commented out and use 10chapters at 100 entries still?

Lastly, the DB-API for createChild is:
function createChild( sourcenode, [name], [type] )

Doesn't that mean that the first argument should be an existing databasenode under which the new child will be placed? I just recently worked on popups for dual classing and adding class nodes so I only used the databasenode.createChild("tag name") which looks more like what you did?

Anyhow - just trying to toss ideas in case helps.
PS - like I said, I haven't made reference pages. Does calling set category MOVE a node from somewhere to under that category or just set another field or property to that category? IF it is "moving" the parent node with like 100 entries in it - it may account for the slowdown although, like you I wouldn't think it would be orders of magnitude over creating them in the first place?

[celestian]

If you commented out the last line: "DB.setCategory(nodeStory, sChapterString);" --> what kind of increase in time are we talking? Also, does it crash if commented out and use 10chapters at 100 entries still?


Node creation in general is not fast. It does still crash with 10 chapters and 100 nodes each even w/o setCategory. It's probably a memory usage issue and not setCategory.



Lastly, the DB-API for createChild is:
function createChild( sourcenode, [name], [type] )

Doesn't that mean that the first argument should be an existing databasenode under which the new child will be placed? I just recently worked on popups for dual classing and adding class nodes so I only used the databasenode.createChild("tag name") which looks more like what you did?


I'm creating story node entries in the root tree "encounter". If I wanted sub-nodes within a entry it would be something like



local nodeChar = DB.createChild("charsheet");
local nodeWeapons = nodeChar .createChild("weaponlist");
local nodeWeapon = nodeWeapons .createChild();
DB.setValue(nodeWeapon,"name","string","Longsword");

[Varsuuk]

Gotcha. I hadn’t thought of the “root node creation” angle. I’ve only in my limited experience always created below another node either via node.createChild(sName) or DB.createChild(parentNode, sName) type thing (off top of my head typing into phone on commute if recall incorrectly)

[Moon Wizard]

The database manipulation is much slower than I would like. One idea that I have been toying with implementing for years is to remove the Lua object tied to each database node, which is where most of the overhead comes from. Unfortunately, this was baked into the FG API years before I was around. So, removing that would break every single ruleset.

Not sure why it's crashing out. Probably something memory related. Is the memory actually growing too large according to Task Manager and crashing the application, or is it just a Lua crash that gets captured?

Regards,
JPG

[Varsuuk]

I’d be happy to try it on one of my PCs. They range from and old 8gb Win10 to desk/laptop 16s and desktop 32gb. I have a Linux with 16 I think and my macbook has 16 for sure.

If it is possible with a command or minimal typing to reproduce, I can try it on diff machines to see if it works on any of them.

[celestian]

The database manipulation is much slower than I would like. One idea that I have been toying with implementing for years is to remove the Lua object tied to each database node, which is where most of the overhead comes from. Unfortunately, this was baked into the FG API years before I was around. So, removing that would break every single ruleset.

Not sure why it's crashing out. Probably something memory related. Is the memory actually growing too large according to Task Manager and crashing the application, or is it just a Lua crash that gets captured?

Regards,
JPG

I'm pretty certain it's memory. I watched the app consume more and more until it crashed when I ran 10 chapters with 100 entries each (went up to around 3.7GB). I ran one (that seems to be about the limit) with 5 chapters of 100 each and just before it ended I got two console messages "Script Error: not enough memory" x2 and it didn't get all the story entries on the last chapter (stopped at 91).

I can use smaller values and it works. 10 chapters with 20 in each 5 separate times and it works. It just doesn't like to do it all at once.

I know you have bigger things on the table right now so if it's an API thing I'll leave it with it's warts for now. It's a very situational need.

[Trenloe]

...went up to around 3.7GB...
Yep, that’s the 32-bit application memory limit on a 64-bit operating system.

[Varsuuk]

Right, thanks Trenloe - forgot - don't matter how much ya got (so to speak), you hit the 32bit limit.

In the future... ;)