sirnoobsauce
April 21st, 2023, 16:09
I recently did a bunch of experimentation and work figuring out a way to run non-blocking code (or as close to non-blocking as I could manage) and figured it might be useful information for the rest of the extension/ruleset development community. This is mainly directed at other folks who work on Extensions/Rulesets if you've ever wanted to have the option to run async code in the style of an executor, or asycnc/await calls.
I will explain the mechanism below, but I put the async code in a github repo in case anyone out there also wants to be able to schedule background jobs in FG extensions or rulesets:
https://github.com/bakermd86/AsyncLoop
I should note that while I did zip this up into an ext file, I don't really think this makes a lot of sense to package as its own standalone Extension. I just did that in case anyone wants to try the included example, but really if you want to make use of this in your own extensions/rulesets, you probably just want to checkout the code from github and include it directly in your own ext/pak file.
Usage is relatively straightforward, although depending on what you want to do it can get as complicated as you want. It works a bit like a map() call in python or JS. You provide a job name, a list of arguments, a target function, and two optional arguments: a callback function and boolean to override the status display setting. Then the iterable table of arguments will be passed to the target function, and any output from those calls is aggregated and passed to your callback function at the end. This is the function signature:
function scheduleAsync(callName, targetFn, callArgs, callbackFn, silent)
The repo includes an examples.lua script showing a simple example where you can run the same operation (multiplying some large number of integers) both synchronously and asynchronously to compare:
...
Comm.registerSlashHandler("asyncMathExample", doMathAsync, "/asyncMathExample <number of args>")
...
function doMathAsync(sCommand, sParam)
local numbers = {}
for i=1,tonumber(sParam) do
local _x = math.random(1, 1000)
local _y = math.random(1, 1000)
table.insert(numbers, { x=_x, y=_y })
end
AsyncLib.scheduleAsync("doMath", wrapDoMath, numbers, mathCallback)
AsyncLib.startAsync()
end
function mathCallback(callName, asyncResults, asyncCount, asyncCpuTime)
Debug.chat("Got " .. asyncCount .. " results in " .. asyncCpuTime .. "s of CPU time")
end
function wrapDoMath(callArg)
return doMath(callArg.x, callArg.y)
end
function doMath(x, y)
local z = x * y
Debug.chat(x .. " x " .. y .. " = " .. z)
return z
end
The argument list (numbers) is a list of tables, and the target function (wrapDoMath) takes those tables as its arguments. The signature for the callback function (mathCallback) is:
callName (str), asyncResults (table), asyncCount (int), asyncTime (float)
callName is whatever you provided, asyncResults is a list of the outputs from your target function, asyncCount is how many input arguments were passed (which can be different to the number of results) and asyncTime is the CPU time of the execution (which can be different to the wall time).
More complex tasks can be achieved using table arguments that contain data on their state (like an instance of a class, although Lua doesn't really have classes but that's a separate topi...). The event loop will check for the presence of the boolean value "isActive" in the input arguments, and if present will re-run the same argument against the target function until isActive is false. This allows for jobs to function like coroutines, with suspended execution and resumption, even though the FG Lua environment does not include coroutines.
An example of this more complex multi-call usecase can be see in the search indexer in my record browser extension:
https://www.fantasygrounds.com/forums/showthread.php?77341-FG-Browser-with-search-CoreRPG-5e-SWD-other
https://github.com/bakermd86/FoogleBrowser
function initIndexer(indexer)
indexer.childNodes = walkChildren(indexer.node)
indexer.isStarted = true
indexer.isActive = true
end
...
...
function updateOnIndex(indexer)
...
...
indexer.isActive = false
end
function runIndexer(indexer)
if not indexer.isStarted then
initIndexer(indexer)
elseif #indexer.childNodes == 0 then
updateOnIndex(indexer)
else
indexNextChild(indexer)
end
end
function newIndexer(node, recordType, isLibrary)
local indexer = {}
indexer.node = node or ""
indexer.recordType = recordType or ""
indexer.nodeType = DB.getType(node)
indexer.nodeStr = DB.getPath(indexer.node)
indexer.isActive = false
indexer.isStarted = false
indexer.isLibrary = isLibrary or false
indexer.isReindex = false
indexer.node_results = {}
indexer.childNodes = {}
return indexer
end
In this example, the tables being used as the arguments are created by the newIndexer() function, and contain state data that is used to chain the execution across across multiple invocations of the event loop. This was necessary because each individual record can take longer to index than would be acceptable in some computing environments.
By including the isActive boolean, the event loop will pass the same table to the runIndexer() function continuously until the indexer has gone through all of its childNodes, and sets isActive to false. I removed all the actual functional code relevant to the indexing itself, this is just showing how a table can be used to perform more complex multi-step tasks using this mechanism.
I will explain the mechanism below, but I put the async code in a github repo in case anyone out there also wants to be able to schedule background jobs in FG extensions or rulesets:
https://github.com/bakermd86/AsyncLoop
I should note that while I did zip this up into an ext file, I don't really think this makes a lot of sense to package as its own standalone Extension. I just did that in case anyone wants to try the included example, but really if you want to make use of this in your own extensions/rulesets, you probably just want to checkout the code from github and include it directly in your own ext/pak file.
Usage is relatively straightforward, although depending on what you want to do it can get as complicated as you want. It works a bit like a map() call in python or JS. You provide a job name, a list of arguments, a target function, and two optional arguments: a callback function and boolean to override the status display setting. Then the iterable table of arguments will be passed to the target function, and any output from those calls is aggregated and passed to your callback function at the end. This is the function signature:
function scheduleAsync(callName, targetFn, callArgs, callbackFn, silent)
The repo includes an examples.lua script showing a simple example where you can run the same operation (multiplying some large number of integers) both synchronously and asynchronously to compare:
...
Comm.registerSlashHandler("asyncMathExample", doMathAsync, "/asyncMathExample <number of args>")
...
function doMathAsync(sCommand, sParam)
local numbers = {}
for i=1,tonumber(sParam) do
local _x = math.random(1, 1000)
local _y = math.random(1, 1000)
table.insert(numbers, { x=_x, y=_y })
end
AsyncLib.scheduleAsync("doMath", wrapDoMath, numbers, mathCallback)
AsyncLib.startAsync()
end
function mathCallback(callName, asyncResults, asyncCount, asyncCpuTime)
Debug.chat("Got " .. asyncCount .. " results in " .. asyncCpuTime .. "s of CPU time")
end
function wrapDoMath(callArg)
return doMath(callArg.x, callArg.y)
end
function doMath(x, y)
local z = x * y
Debug.chat(x .. " x " .. y .. " = " .. z)
return z
end
The argument list (numbers) is a list of tables, and the target function (wrapDoMath) takes those tables as its arguments. The signature for the callback function (mathCallback) is:
callName (str), asyncResults (table), asyncCount (int), asyncTime (float)
callName is whatever you provided, asyncResults is a list of the outputs from your target function, asyncCount is how many input arguments were passed (which can be different to the number of results) and asyncTime is the CPU time of the execution (which can be different to the wall time).
More complex tasks can be achieved using table arguments that contain data on their state (like an instance of a class, although Lua doesn't really have classes but that's a separate topi...). The event loop will check for the presence of the boolean value "isActive" in the input arguments, and if present will re-run the same argument against the target function until isActive is false. This allows for jobs to function like coroutines, with suspended execution and resumption, even though the FG Lua environment does not include coroutines.
An example of this more complex multi-call usecase can be see in the search indexer in my record browser extension:
https://www.fantasygrounds.com/forums/showthread.php?77341-FG-Browser-with-search-CoreRPG-5e-SWD-other
https://github.com/bakermd86/FoogleBrowser
function initIndexer(indexer)
indexer.childNodes = walkChildren(indexer.node)
indexer.isStarted = true
indexer.isActive = true
end
...
...
function updateOnIndex(indexer)
...
...
indexer.isActive = false
end
function runIndexer(indexer)
if not indexer.isStarted then
initIndexer(indexer)
elseif #indexer.childNodes == 0 then
updateOnIndex(indexer)
else
indexNextChild(indexer)
end
end
function newIndexer(node, recordType, isLibrary)
local indexer = {}
indexer.node = node or ""
indexer.recordType = recordType or ""
indexer.nodeType = DB.getType(node)
indexer.nodeStr = DB.getPath(indexer.node)
indexer.isActive = false
indexer.isStarted = false
indexer.isLibrary = isLibrary or false
indexer.isReindex = false
indexer.node_results = {}
indexer.childNodes = {}
return indexer
end
In this example, the tables being used as the arguments are created by the newIndexer() function, and contain state data that is used to chain the execution across across multiple invocations of the event loop. This was necessary because each individual record can take longer to index than would be acceptable in some computing environments.
By including the isActive boolean, the event loop will pass the same table to the runIndexer() function continuously until the indexer has gone through all of its childNodes, and sets isActive to false. I removed all the actual functional code relevant to the indexing itself, this is just showing how a table can be used to perform more complex multi-step tasks using this mechanism.