Batch or delay node generation

[mattekure]

I am working on a Syrinscape CSV import process that will be creating several hundred story entries. The problem I am running into is that trying to process the data and create the story entries quickly uses up too much RAM for FG Classic. It tends to crash after 100-200 entries.

Is there any way I can batch or delay these node creations so FG has the chance to release the memory of the previously created ones?

[Moon Wizard]

There's no delayed creation of nodes. Also, the triggers for node events happen synchronously inline, so there should be no "temporary" memory situation. My guess is that it's not a temporary memory issue; but not sure without digging into code.

Might be a stack overflow (Lua and/or application) based on how you are building, if I was going to guess...

JPG

[mattekure]

This is the code I wast testing with. I have a window control where the user specifies the number of entries to create. When the button is clicked it calls this function to create the nodes.

local nEncNode = DB.findNode("encounter");
local aNode = DB.findNode("MK.numToCreate");
local s = aNode.getValue();
if tonumber(s) ~= nil then
numToCreate = tonumber(s);
else
numToCreate=1;
end
while numToCreate > 0 do
local nNode = nEncNode.createChild("test"..numToCreate);
local nNodeName = nNode.createChild("name", "string");
nNodeName.setValue("MK "..string.format("%03d", numToCreate));
local nNodeText = nNode.createChild("text", "formattedtext");
nNodeText.setValue("newtestvalue");
numToCreate = numToCreate -1;
end

So really, its just running a for loop and creating the nodes one at a time. Is there a more efficient way to create a series of nodes?

When I run this code for 100-200 entries, FG hits the RAM limit and crashes.

[Trenloe]

A couple of different things to try (I don't know if either of these will work).

1) Declare nNode, nNodeName and nNodeText as local variables before the While.. Do loop, then set them within the loop. I don't know if declaring local variables over and over again might be adding to the issue.

2) Try using the DB package (https://www.fantasygrounds.com/refdoc/DB.xcp) more instead of creating 3 database node objects. It's not always possible, but try to minimize the use of database node objects if you're experiencing memory issues.

[mattekure]

I redid my code based on your suggestions and can now process around 500 before it crashes, so thats some improvement. This is what it looks like now.



local sNodeName = "";
local sNodePrefix = "encounter.";
local sNameText = ".name";
local sTextField = ".text";
local sNodePath = "";
while numToCreate > 0 do
sNodeName = "test"..numToCreate;
sNodePath = sNodePrefix..sNodeName..sNameText;
DB.setValue(sNodePath, "string", "MK "..string.format("%03d", numToCreate));
sNodePath = sNodePrefix..sNodeName..sTextField;
DB.setValue(sNodePath, "formattedtext", "new test value");
numToCreate = numToCreate -1;
end

[mattekure]

I just test this exact same code in Unity, amazing difference. I created 600 story entries. It took about 5.5 minutes but RAM usage never went above 1Gb.

[darrenan]

Have you thought about doing this outside of FG? It would be pretty easy to import directly into the db.xml from a dedicated program or script.

[celestian]

I've tried various things to resolve this issue under FGC ... even adding a sleep after X processes. I think the only way to actually "fix" it is to process X amount, store the rest of the text in a buffer, prompt the user for "next" batch and then restart.

The only reason I've not tried the last method is... FGU doesn't give me this problem so... I've not tested but I'm pretty sure it would work ... assuming I properly understand the issues causing it.

[mattekure]

Have you thought about doing this outside of FG? It would be pretty easy to import directly into the db.xml from a dedicated program or script.

Actually yeah. My current syrinscape sound module was done exactly this way. I use a python script to run through the data and output a db.xml which is packaged in a module.

The reason I am looking into this is based on some comments by the Syrinscape developers. They recently made a CSV export capability, which exports a file with all the links to sounds that you are authorized. So when I download my CSV, it has only the links I have access to. I have their "Sypersyrin" subscription so I have access to all sounds except ones that people create themselves via their soundset creator, so I can create a module with everything. However, for a user who doesnt have the same subscription, they will have a bunch of dead links. Also, anyone who creates their own stuff wont be able to get links to it.

So, my solution is to build a CSV importer to import that 22K line CSV file, and generate the soundlinks internally. This will allow users to update it whenever they want and will give them soundlinks to all the sounds they own, including any they created themselves.

[Trenloe]

So, from what you've seen, does it look like a limit within a single function/process "space" running our of memory?

I'm wondering if you can package it up into chunks like celestian mentions, but instead of prompting the user, kick off a new batch using OOB messaging. This might be similar to starting a new process "space" and release the memory used in the previous processing. I have no idea if this would work or not.

[mattekure]

With the limited testing, it does seem like it might be limited within a single function/process space. But I've noticed, at least with FG Classic, that if I do a bunch of nodes, like 100 or so, the memory is not release. So it will balloon up to 2.5Gb, and when the process completes, it stays in use. In Unity, it never rose anywhere near as high, and went right back down once the process completed. So I agree with MoonWizard that it may be a stack overflow somewhere.

It might be possible to chunk it. The easiest way I can think of is to load all the data into memory without creating the nodes. Store it in a global variable somewhere. Then in a processing window have a button that says "Process the next 100 records" or something. Perhaps include a countdown for the number of lines left to process. This would require the user to have to click it multiple times, but thats better than having it crash or hang.

[bojjenclon]

You mention it seems the memory isn't being released - I don't know exactly how FG sandboxes its Lua calls, but is it possible to manually trigger garbage collection (with the aptly named collectgarbage() function) after processing X number of entities? It might create a temporary CPU spike but if the issue is memory usage it might be worth the extra cycles. It could require some tweaking, such as setting values to nil prior to invoking the GC. I don't know specifically how Lua's GC model works, but it might not invoke its collection of resources until your while loop finishes - or worse, until the function completes and the local variables are no longer referenced - so forcing it to work part way through could be an option.

[celestian]

I've tried various things to resolve this issue under FGC ... even adding a sleep after X processes. I think the only way to actually "fix" it is to process X amount, store the rest of the text in a buffer, prompt the user for "next" batch and then restart.

The only reason I've not tried the last method is... FGU doesn't give me this problem so... I've not tested but I'm pretty sure it would work ... assuming I properly understand the issues causing it.

So this turned out to be pretty easy to setup with my current import process. It does seem to work. In short, I count each line I process, when I reach XX number I start storing the remaining entries and do not process them. After the loop (stepping through each line) I check to see if "aImportRemaining" is > 0 and if so I place the remaining process "text" (no longer has the entries we processed) into the text field in the import window and prompt the user to press IMPORT again for the next batch.

It's a bit tedious but it gets around the whole memory issue thing.

This would be better handled using a external python/perl script to generate the data due to the size of the import block but normal people aren't going to have that laying around. This will be the "simplest" for most novice users, it'll just take a while to finish.

The code I used to do this will be available in a repo once the project is ready for public view (expect this weekend).