Batch or delay node generation

[mattekure]

I am working on a Syrinscape CSV import process that will be creating several hundred story entries. The problem I am running into is that trying to process the data and create the story entries quickly uses up too much RAM for FG Classic. It tends to crash after 100-200 entries.

Is there any way I can batch or delay these node creations so FG has the chance to release the memory of the previously created ones?

[Moon Wizard]

There's no delayed creation of nodes. Also, the triggers for node events happen synchronously inline, so there should be no "temporary" memory situation. My guess is that it's not a temporary memory issue; but not sure without digging into code.

Might be a stack overflow (Lua and/or application) based on how you are building, if I was going to guess...

JPG

[mattekure]

This is the code I wast testing with. I have a window control where the user specifies the number of entries to create. When the button is clicked it calls this function to create the nodes.

local nEncNode = DB.findNode("encounter");
local aNode = DB.findNode("MK.numToCreate");
local s = aNode.getValue();
if tonumber(s) ~= nil then
numToCreate = tonumber(s);
else
numToCreate=1;
end
while numToCreate > 0 do
local nNode = nEncNode.createChild("test"..numToCreate);
local nNodeName = nNode.createChild("name", "string");
nNodeName.setValue("MK "..string.format("%03d", numToCreate));
local nNodeText = nNode.createChild("text", "formattedtext");
nNodeText.setValue("newtestvalue");
numToCreate = numToCreate -1;
end

So really, its just running a for loop and creating the nodes one at a time. Is there a more efficient way to create a series of nodes?

When I run this code for 100-200 entries, FG hits the RAM limit and crashes.

[Trenloe]

A couple of different things to try (I don't know if either of these will work).

1) Declare nNode, nNodeName and nNodeText as local variables before the While.. Do loop, then set them within the loop. I don't know if declaring local variables over and over again might be adding to the issue.

2) Try using the DB package more instead of creating 3 database node objects. It's not always possible, but try to minimize the use of database node objects if you're experiencing memory issues.

[mattekure]

I redid my code based on your suggestions and can now process around 500 before it crashes, so thats some improvement. This is what it looks like now.

	local sNodeName = "";
	local sNodePrefix = "encounter.";
	local sNameText = ".name";
	local sTextField = ".text";
	local sNodePath = "";
	while numToCreate > 0 do
		sNodeName = "test"..numToCreate;
		sNodePath = sNodePrefix..sNodeName..sNameText;
		DB.setValue(sNodePath, "string", "MK "..string.format("%03d", numToCreate));
		sNodePath = sNodePrefix..sNodeName..sTextField;
		DB.setValue(sNodePath, "formattedtext", "new test value");
		numToCreate = numToCreate -1;
	end

[mattekure]

I just test this exact same code in Unity, amazing difference. I created 600 story entries. It took about 5.5 minutes but RAM usage never went above 1Gb.

[darrenan]

Have you thought about doing this outside of FG? It would be pretty easy to import directly into the db.xml from a dedicated program or script.

[celestian]

I've tried various things to resolve this issue under FGC ... even adding a sleep after X processes. I think the only way to actually "fix" it is to process X amount, store the rest of the text in a buffer, prompt the user for "next" batch and then restart.

The only reason I've not tried the last method is... FGU doesn't give me this problem so... I've not tested but I'm pretty sure it would work ... assuming I properly understand the issues causing it.

[mattekure]

Have you thought about doing this outside of FG? It would be pretty easy to import directly into the db.xml from a dedicated program or script.

Actually yeah. My current syrinscape sound module was done exactly this way. I use a python script to run through the data and output a db.xml which is packaged in a module.

The reason I am looking into this is based on some comments by the Syrinscape developers. They recently made a CSV export capability, which exports a file with all the links to sounds that you are authorized. So when I download my CSV, it has only the links I have access to. I have their "Sypersyrin" subscription so I have access to all sounds except ones that people create themselves via their soundset creator, so I can create a module with everything. However, for a user who doesnt have the same subscription, they will have a bunch of dead links. Also, anyone who creates their own stuff wont be able to get links to it.

So, my solution is to build a CSV importer to import that 22K line CSV file, and generate the soundlinks internally. This will allow users to update it whenever they want and will give them soundlinks to all the sounds they own, including any they created themselves.

[Trenloe]

So, from what you've seen, does it look like a limit within a single function/process "space" running our of memory?

I'm wondering if you can package it up into chunks like celestian mentions, but instead of prompting the user, kick off a new batch using OOB messaging. This might be similar to starting a new process "space" and release the memory used in the previous processing. I have no idea if this would work or not.