Is it feasible to develop scripting API's for JavaScript or C# along side Lua APIs?

[StoryWeaver]

After reading a number of threads on how Lua may be holding back overall performance for FGU in some cases.

Due to things such as offering no multi-threading, and that adding multi-threading to Lua would take to long to be on the table for the developers. Partly as the current commercial Lua solutions for that are not mature enough.
General slower running of Lua compared to some more modern languages, etc.

I understand that keeping support for Lua to allow for the running of the colossal catalog of available products and rulesets is a must, and that's all good.

I am more wondering about the way ahead, for future products, modules, extensions, rulesets, ruleset updates, expansions.

So as the title says, would it be feasible to develop scripting API's for JavaScript or C# along side Lua current APIs?
To allow for all of the mentioned above to be coded in JavaScript or C# (compiled), connecting to the same core API functionality that is now offered for Lua.
I imagine it's operating through some API middleware to the C# FGU core product functionality. So to add a similar layer for JavaScript and/or C# is what I'm pondering. How much work that would be and if it is something to consider by SmiteWorks?

Of course only after FGU has been released and had some time to mature.

I am aware CoreRPG is written in Lua, and the other major rulesets use that and build on top with more Lua. CoreRPG calls on API's though (as documented on the developer wiki) much as all others. So that's the part I'm wondering about.

I'm not suggesting these core components be rewritten in say JavaScript anytime soon at least.
But the first step for such changes would be to unlocking that potential with API middle-layers I'd imagine.
Then there might be some interest from the community to rework some of these rulesets to more modernized languages, and community scripters and coders could take advantage of the ever developing language improvements, tools and utilities as they come along in their work.

Recent threads on Lua started this line of thought again, and I know the devs have enough to work on for the next year no doubt as it is *laughs*. Software is never complete, a blessing and a curse.
So just throwing some thoughts out there. It would be great if something like this would eventually happen to help with some of the performance related issues and a quality of life improvement.
I know that JavaScript for the scripting side of things has and is being actively used for some other VTT's quite effectively, and having a wider pool of people who already know how to write JavaScript compared to Lua can't hurt to bring more community developers to bear.

In the end the devs for FG will know best if this is even something that is feasible to attempt. These are simply my own ruminations and wishful thinking on the subject. I recall probably some half a year ago asking something similar, retaining more to writing more extensive plug-ins to improve FGU more in C#. At the time it was not something that was likely to happen. Mind you compiled packages would be easier sold as DLC's over Steam for example. Could package new Unity coded effects, complete interface overhauls and all sorts of things, but that's another subject. :)

ps. Which reminds me, the Workshop tools on Steam would be great for managing community extensions and automatically keeping them up to date!

That aside, my compliments to the dev team for how hard they've been working on FGU in general and the speed they're tackling the more prominent recent unexpected issues that have come up due to the unforeseeably large influx of new FG roleplayers. :)

[pollux]

Let me preface this by saying that I'm not a dev at Smiteworks and I have no special knowledge of their plans.

First off, lua is not a bad scripting language. It's taking some heat here because it's tied up in some important performance bottlenecks for us... but the problem isn't that it's a bum technology stack. For context, Javascript was also single-threaded for decades and only progressed beyond that due to the dogged investment of hundreds of millions of dollars from giants like MS and Google... and it only BARELY evolved beyond it at all. Web-workers still have all kinds of overheads and limitations that make them appropriate only for certain use-cases and nodejs is still mostly dominated by single-threaded code using async APIs. Python, another top-5 most popular language has had its developer community discuss the global-interpreter-lock and how it prevents efficient multi-CPU computation for years and been unable to address the problem successfully. Fantasy Grounds is 16 years old, and it's not at all weird for systems of that vintage to have hard threading challenges. Lua, the language, is not the cause of these problems. And switching to another language is not a viable path to fixing the problems.


I am aware CoreRPG is written in Lua, and the other major rulesets use that and build on top with more Lua. CoreRPG calls on API's though (as documented on the developer wiki) much as all others. So that's the part I'm wondering about.

This is your key insight, and the core from which many other important insights trace out.

I think you're imagining a model where FGU is the TTRPG engine, and people add scripts on top of it to chain together sequences of TTRPG actions that are already built in FGU. You're imagining that most of the hard work is done in FGU itself and that the Lua code has a simple job that it's doing slowly due to language limitations. This is a common model in video games with a lua scripting engine, and it's not crazy to imagine it... Roll20 macros do work in more or less this fashion. But it's also not what FGC/FGU looks like. FGC/FGU are the lua scripting engine, plus a library of ttrpg primitives like dice, plus the I/O that allows the lua scripts to interact with people, networks, disks, and displays... FGC/FGU itself isn't a whole lot more. The TTRPG engine(s) are the rulesets, and they're written in lua. There are twenty official rulesets, plus countless more written by the community. There are around 1500 pieces of official content that run on top of those rulesets. Without the lua rulesets, FGC/FGU have no meaningful concept of a character sheet, an NPC statblock, an item, armor-class, saving-throws, etc. This isn't Roll20 where you would just lose your "macros", it's macros pretty much all the way down.

So... ugh... that's a drag, right? Lua is single-threaded and Lua is FG, how are we ever going to get decent performance? The future may not be so bleak, though:


FGC performance isn't so bad. And that used a single-threaded Lua engine too, so there's prior art that shows it's possible to make this work pretty well.
There must be some new inefficiencies at work that apply only to FGU making things slow-ER than they are in FGC. If one of these inefficiencies turns out to be widespread, there may be hope for an update that reduces the cost of some common action (maybe speeds the serialization of Lua data-structures to C# types or vice-versa), a small number of optimizations might have a really big impact if they're in code that is called hundreds of times per second during a pause. We don't know yet if there are big wins like this because the dev team haven't turned their eye to performance yet. As important as performance is, they've been dealing with EVEN MORE IMPORTANT STUFF like people being unable to install the software due to updater overload and people unable to connect to games due to networking problems.
But maybe there are no inefficiencies making Lua slow in FGU, maybe Lua is just spending too much time computing stuff. Even still, the long pauses might still be dominated by a small amount of foundational Lua code. For example, maybe lua is being used to parse XML (I don't know if it is or not, but let's pretend). Maybe instead FGU can parse the XML in C# more quickly, and hand Lua a representation that's more efficient to handle. Maybe there are only a few utility functions used to parse XML in the rulesets, and by making them 10x faster we can make performance acceptable. Again, because the devs haven't dug in yet... we don't know if there are big wins to be made in a small number of places by doing work in smaller chunks (so the UI can process some stuff between batches), by optimizing the Lua code so it runs faster, or by offloading heavy lifting to multi-threaded C# code and waiting for a result.
But maybe life is hard. Ugh. Maybe there are no big easy wins either in the interfaces between Unity and Lua or in the Lua itself. The next thing I would think about would be adding NEW async APIs that Lua code can opt into. APIs that let Lua ask for some heavy work to happen in Unity, but the Lua code doesn't just sit around doing nothing while it waits for an answer. It can do other stuff and get notified when the answer is ready so it can come back and deal with the result. This kind of interface might let us open up the NPC list and see a "loading" message instead of having the whole UI pause. Or it might let us see NPCs get added in batches of 10 rather than freezing until the whole list is complete. If it comes to this, it will be a long slog. Rulesets will need to change in ways that are incompatible with FGC and the changes are not likely to be well localized. But async APIs are proven ways to scale single-threaded runtimes, it's a big part of what drove nodejs's popularity in the javascript community.


So, I do get people's frustration. And I admire the desire to generate dialogue to help improve the situation. But mostly I think we need to trust the devs for a while more. I'm sure that nothing I've described above is news to them, they just need time to implement it. Is it frustrating that FGU isn't farther along, now that we're past the targeted release date? Of course it is. But Smiteworks is a small engineering team. When I look at what they left behind from FGC and what they kept and brought forward to FGU... from the outside... it looks like a very smart team pulling forward only the pieces they need and replacing all the solved problems with third-party libraries. It all looks very smartly handled. There are still legacy problems, but you can't expect to leave ALL the legacy behind and keep what's good about the old thing... you can only be smart about pulling forward legacy problems that "pull their own weight" by providing lots of value. And the lua engine... in spite of whatever performance problems it has... is worth its weight in platinum because of the ruleset code it supports.

[StoryWeaver]

That was a good thought out response. :)

I can't say I disagree with anything you've said, all of these things take a lot of time and I think the devs are overall doing a fantastic job.
Always worth bringing some thoughts to the table.

It will be interesting to see if they'll be able to offload some of the work onto multi-threaded C# operations over time from the Lua scripts, and implements some async API's that Lua could take advantage off.
I know they've been having some look into the latter at least.
This would hopefully give a smoother experience and eventually eliminate those instances of FGU and FG freezing while there's some heavy processing in the background (noticeably HDD things, though after getting a SSD this has become much better I must say). But it's not something you notice in most software, you can usually always interact with other parts to some extent in the meantime.

Though this is all something to come hopefully, and performance is going to be at the last stage before release, so patience is key.
Sometimes I can get overly passionate about FG and what I want it to be and become. *laughs*

Interestingly to FG's and FGU's credit. I mixed in another highly popular VTT for a one shot over the weekend which I've used extensively in the past.
Over the whole session I couldn't help but notice how awkward, ill organized and slow everything felt, and how starkly better and more fun it was to play in FG.
It was like my hands were tied and going back to the stone age of VTT's after becoming spoiled by everything I've become used to with FG, despite its own flaws!
Worlds apart, despite its popularity. People who haven't tried FG simply do no know how much better a VTT can be, particularly from the DM side of things.

The vast majority of players I invite to try a game with me in FG end up loving it after a couple of sessions or so.