Ruleset Automation

[LordEntrails]

There has been a fairly involved discussion regarding ruleset automation over in the 5E Bugs thread (https://www.fantasygrounds.com/forums/showthread.php?49201-5e-Bug-Reports-(Part-5)&p=470770&viewfull=1#post470770). It has been suggested that it be moved to a new thread. I've copied a little of the discussion below. Please continue the discussion here :)


No, the fact that things don't have effects attached has nothing to do with the way modules are constructed.

Included in the ruleset code is a parser which looks for certain words and phrases and then tries to create effects from that. So, for example, if the text of a spell or ability or whatever has the phrase 'and the target is paralyzed for 1 minute' then FG will recognise that phrase and create an effect of [Paralyzed][1 MIN] when that thing is dragged into the actions tab or if it's an NPC is put onto the Combat Tracker. This is true for pretty much everything, abilities, NPCs and spells.

Now this system generally works provided that the wording or phrases used are consistent. So, for example, in an NPC stat block the attack actions are almost always <type> Weapon Attack: +x to hit. Hit: ndn <type> damage. So FG can translate that into the necessary effects when the NPC is placed on the CT. Other examples include the skills choices for a class, saving throw types and damage from spells and many other things. All of these things generally are worded consistently. See the wiki here (https://www.fantasygrounds.com/wiki/index.php/5E_Effects#Automating_Effects_on_NPCs) for more examples of what FG can pick out for NPCs specifically and this post (https://www.fantasygrounds.com/forums/showthread.php?48739-5e-Class-Background-Race-Feat-interpreted-strings) for other things that FG can pick out of the wording used in backgrounds, classes and races.

A quick glance at magic items will tell you that there are a hundred ways that those are worded and each would need to have specific words looked for in order to create the effects. In the example we quoted earlier the Ring of Protection FG would need to interpret 'you get a +1 bonus to AC and saving throws whilst wearing this ring'. This is what I meant by automating everything given sufficient development time. This phrase and variations of this phrase would need to be programmed into FG so that the correct effect could be created - and only when the item is worn. You might find Celestian's extension here (https://www.fantasygrounds.com/forums/showthread.php?40833-5E-Advanced-Effects-(items-npcs-characters)) useful - you still need to create the effects yourself but it will apply the effect only if the item is equipped by the player.

I could be wrong here but I am pretty sure that FG doesn't look at the wording of items at all (apart from weapons and armour which are special cases); be they magic or otherwise simply because there is just too much uniqueness and variation in the wording to have any chance of creating proper effects.

As I said earlier the level of automation in 5e is unsurpassed although Celestian's 2e ruleset and Trenloe's Pathfinder 2 ruleset are very close rivals (and Starfinder is catching up too I think), but currently it is too much to ask that everything be automated given the hours that would need to be put into it in order to achieve that level.


Cool, that's all great info, and I'm glad this issue isn't limiting the automation. I agree that the language parsing and automation that does exist is spectacular, and is one of the biggest selling points of FG to me. Simply put, as HuseyinCinar said, it's disappointing that some of the coding that is clearly possible doesn't make it into the official product, but that is clearly a topic of conversation worthy of a thread of it's own, and I'll stop monopolizing this one :p. I just wanted to put in my 2c that I think the issue is worthy of discussion and further inspection. As much as I hate to be one of "those people" on the forums, I (and we all) want FG to be as incredibly awesome as possible. I think it's already great, FGU is going to make it amazing, and I can't wait to see what special projects are going on behind the scenes. The squeaky wheel gets the grease though, and I think this spot needs greasing ;). Despite the fact that I clearly don't know a whole lot about how the sausage gets made, I had noticed issues like this in the past and wanted to chime in that I thought it was worth fixing. If nothing else, I'll be happy to know that the next time it comes up, it will no longer be the case that "it's never been a problem before."

Thanks very much for your thoughtful responses! They are much appreciated.

[LordEntrails]

One point I wanted to add to the discussion of ruleset automation, and in particular adding phrases to the parser are the following 2 points that help to point out that automating a ruleset (in particular D&D 5E, but applies to any ruleset) is that adding more phrases to be parsed (such as for spell effects etc) has two major draw backs.

1) The more phrases that have to be parsed, the harder maintaining the code becomes. Things like conflicting phrases becomes a more challenging issue. Updates to the ruleset take longer, and therefore improvements to the software will come more slowly. In short, the more automation we have, the longer fixes and enhancements will take to roll out.

2) Performance. The more phrases that have to be parsed when a creature is added to the Combat Tracker, the longer it will take FG to parse and prepare the CT. I believe their are already reports in the Pathfinder Ruleset of measurable delays when a significant number of NPCs are added to the CT. As the software has to do more processing, the slower the software will become.

Neither of these are things that are hard lines. They are incremental costs that will be incurred as automation increases.

[celestian]

I've danced around this myself for a while. There are ways to make it a bit more user friendly (I find string parsing not very user friendly myself) but to do that you give up performance by adding various and many nested nodes in the data.

The more nodes you have the slower things load into the CT. It's one reason I had to re-write the DM side of the CT for the AD&D ruleset. Without the re-write the load times were very dramatic.

With the extra bits I was able to add new effects and automation features I wanted. All of the spells and bulk of the items have effects, attacks/powers pre-configured. I was also able to automate equipped items/effects and racial/class effects.

Outside of that there are some other things that can be automated. Improved AC versus ranged or melee. Flat ability score change. Mirror image, stoneskin and specific types of damage absorb/temp hps (tho I am not sure 5e has temp hp for something like fire).

What type of additional automation are you looking for?

[Trenloe]

The 5E ruleset does a lot with word parsing - which has a good success rate as WotC products generally use repeatable wording. For some of the base entries that don't work, there has been some hard coded automation in the base ruleset to carry out automation based on the name of the record.

For Pathfinder Second Edition we don't have the luxury of repeatable wording in most cases - feats, spells, class abilities. Some things (initial class data) is fairly repeatable and basic string parsing can be used, but it's rare.

So, for PFRPG2, I've started experimenting with an automation field in certain records. This is hidden by default, which can be changed through the campaign options. Some information on what I've done so far: https://www.fantasygrounds.com/wiki/index.php/PFRPG2_Experimental which basically allows for database fields in the PC charsheet to be directly modified/set - of use only for static DB fields, but it's a start. Release 12 tomorrow will add defense proficiency automation and a base special automation handler. I plan to expand these over time. The issue with this is that it adds a lot of time to creating DLC content - having to write and test the automation field for every ability/feat. And, as usual, it will probably be impossible to cover all possibilities. I doubt I'll ever do this for every single record in DLC, but will probably keep to common feats/abilities - where I go and how successful it will be remains to be seen...

[Moon Wizard]

The approach that Trenloe is taking is one of the things I've been keeping in my head for quite a while. Basically, you come up with a way to provide extra fields that allow you to automate attacks/damage/effects and more, and move away from trying to read the text. By allowing people to fill out those fields, it gives finer grained control, easier to maintain and less conflicts.

However, increasing the number of fields to be copied and instantiated actually increases the UI complexity and database overhead. This is the fine line that celestian has been walking, since his automation is arguably even more detailed than 5E. The cost is in the UI performance overhead and database node copying overhead. (Parsing some text in Lua is fairly tame in comparison for performance.) Thus, why celestian has been having to reconfigure UI (among other reasons).

If we had time to actually redo this (which we don't); I'd probably move to some sort of coding system similar to effects where you can have a single database field lay out all the "automation actions" for each power/spell/ability/attack. Then, we could build the fields dynamically as needed in the UI without even tying to the database. But, this is just a prototype concept that I've been bouncing around in my head, since we're so focused on FGU right now.

One of the challenges with changing anything (especially in 5E) is that you have to update the entire catalog of DLC to work with your changes. So, it's actually easier to change this assumption earlier in the ruleset coding process.

Cheers,
JPG

[esmdev]

I find the word parsing for 5E to be cool and fairly easy to read. The only real problem that I have with it is the lack of specifics on how to format the statement to get what result. I asked once, long ago if there was a list and the answer was basically no. I think it would do some good if there was a master list.

I resisted creating new entries in the NPC section for months because using the PC sheet was easier than trying to figure out exactly how to format special features and such. I've gotten better at it but I'm no master. I just last week figured out that I needed to have a separate entry for innate spellcasting in addition to spellcasting.

It would be nice if miscellaneous items could apply various effects. Like gauntlets of ogre power adjusting strength on equip and unequip.

Trenloe I like your setup but it, like effects, is arcane to the point it is a barrier to some. I think it is a solid direction but please give lots of examples of how to format stuff.

One thing that would also help 5E (and 2E and really every ruleset) is a stop before applying effect or damage in the CT like the one found in SWADE. It would help when people accidentally double roll damage, have feats that let them reroll damage on 1 or 2, accidentally stack sneak attack effects, etc. It wouldn't be a big deal if 5E didn't stop at 0 and recorded the full damage so you could reverse the damage and just apply but if you don't happen to know exactly how many hits something had when an accidental double roll takes place it's super hard to undo. That sort of feature would help with automating a lot of conditionals that show up here and there.

[Zacchaeus]

I find the word parsing for 5E to be cool and fairly easy to read. The only real problem that I have with it is the lack of specifics on how to format the statement to get what result. I asked once, long ago if there was a list and the answer was basically no. I think it would do some good if there was a master list.

For NPC's see here https://www.fantasygrounds.com/wiki/index.php/5E_Effects#Automating_Effects_on_NPCs
For classes, backgrounds, feats, and races see here https://www.fantasygrounds.com/forums/showthread.php?48739-5e-Class-Background-Race-Feat-interpreted-strings
For other information on effects see here https://www.fantasygrounds.com/forums/showthread.php?41478-Effects-Videos-for-5E and here https://www.fantasygrounds.com/wiki/index.php/5E_Effects

Also you can reverse damage by holding CTRL and dragging the damage from chat back onto the creature that took the damage.

[esmdev]

Thanks, I had figured most of it out from the wiki, but that section on class, race, background should really be in the wiki. That would make it much easier to find.

I will definitely try the control-drag. That will help a lot.

[Neovirtus]

Thanks for making the thread, LordEntrails.

Anyone who's unclear of what is being discussed can follow the link back to the original thread, but suffice to say we got into the weeds a little bit on why some items don't quite "work out of the box." Zacchaeus made some very illustrative posts about how the par5e parser and the limitations it has, which I won't mangle trying to simplify here for discussion. In the end though, I think I'm still unclear on exactly why the two cases we discussed are different: a suit of Mithril Plate Armor and a Ring of Protection.

An item of Mithril Plate armor, or really any other variant-type item is simply reskinned by the parsing process, so you get Plate armor that's called Mithril Plate, and a note that it shouldn't cause disadvantage on stealth checks. In order for a different, "correct" Mithril Plate item to be included in a module, there would have to be a separate, Plate armor base item that had that correct property built into it. I guess Mithril plate isn't in the SRD so it can't be the "base item"? I'm still confused by this, but I take Zacchaeus at face value that this is the way it must be.

Alternatively, a Ring of Protection +1 is basically delivered in the module as an item, with the text copied from the module. It does nothing when equipped to a character without player/DM intervention. But this is not a limitation of the module creation process (as above) but rather the fact that item descriptions are not parsed for effects at all, and if they were (and the parser were "taught" how to read the most common effect phrasings) it would significantly increase the calculation load.

So... It is certainly a slippery slope of where to draw the line as far as expectations for parsing text. I try to view these things as a collaborator, and be understanding that there are technical limitations that prevent things from being perfect. And that if these things were included, developer time would increase, updates would come slower, things would likely cost more, etc. I get it.

But as a customer who prior to yesterday didn't understand how a module is made (and still has a rudimentary understanding at best!) it is frustrating when a simple magic item that's been around for decades isn't coded. My customer brain thinks: "I'm paying for module X, I paid for FG, the module should include everything in the physical book, coded for FG." To me that includes automation, as automation is part of FG... I know that might be unreasonable, but it is what it is. I still think FG is great in it's current form, but this is a point of pain for me. Whenever my current party in Curse of Strahd, or in a homebrew with items from the PHB and DMG (which I paid for), gets a parcel, there is then a 5 minute discussion of how to get it coded. It sucks. I think the bar should be higher for a published module/handbook. Please discuss.

[Trenloe]

Let's look at a little bit of FG history... Once upon a time, FG provided the tools for players and GMs to automate their characters and creatures up to the level that they wanted. It was up to them to apply the effects, data, etc. as their characters were created and evolved over time. Any Smiteworks provided data that was available (OGL or commercial) was pretty much informational (with the odd exception) and players and GMs had to set things up how they wanted - depending on the level of automation they were looking for.

Then, with the advent of D&D 5E (even before FG got the license) someone decided it would be cool to code for more automatic automation - helped by the repeatable text used by WotC. This was great and a lot of people use it and enjoy it, but there will always be exceptions and how these are handled is up for discussion. As a RPG product matures and brings out more material the number of these exceptions increases. Fantasy Grounds has, quite sensibly, gone for the 80/20 rule - trying to provide some level of automation for 80% of cases, the other 20% of cases need to be handled manually. In some RPG system, with non-repeatable data, even 80% is a big challenge (see PFRPG2, for example). This is a realistic approach as the 80/20 rule is usually reversed - i.e. to do the last 20% will take 80% of the work. If this final 20% was tackled, then there would be a lot of other things FG users are asking for that won't be done. Smiteworks is a small company - everything is a balancing act of effort vs. reward.

As an outsider to the background, code and FG development processes, would I want to see near 100% automation of abilities, feats, items etc. within FG? Yes, of course I would!

However, being an insider, and being realistic (that's what we need to be here, realistic), there has to be a line drawn somewhere as it is not commercially viable for Smiteworks to provide a very high level of automation. This is a niche, within a niche and even for D&D, which far outsells and is played more than any other RPG system on FG, spending 80% more time on 5E products (that have a lot of data needing automation) to provide a high level of automation is simply not financially viable. And trying to do the same for other FG, non 5E D&D products, would lead to the development of these products (done by community developers) to pretty much dry up.

Like I say, in a perfect world I'd love to see it, but I imagine most of us want Smiteworks to be successful and continue to get licenses for, and develop products for, all sorts of RPG systems - not just the massively popular one. So, like I say, there needs to be an accepted level. Sure, there will always be exceptions to these - like the couple of items mentioned above, maybe the ring of protection should be coded, and maybe mithril plate should be (just an example).

In the end, if it takes 5 mins for everyone to get FG setup the way they want it when they get a bunch of new items, is this such a big hassle? I'm sure to some it is. But for most, spending this time to save more than this time in future is worth it. And all of use like different levels of automation - there's been threads in the past asking "how much automation do you want" and the results were widely spread out, so what is hard coded for most, might not work for others. I agree that some automation would be simple enough (ring of protection, for example) - unless there's another item, or some ability, that affects the base effects of that item - then we have an exception that needs to be coded manually. And if just one thing needs to be coded manually, then we encounter this expectation of "I'm paying for module X, I paid for FG, the module should include everything in the physical book, coded for FG." I agree that there is this expectation there - and this maybe needs to be managed better.

It's good to see that various devs (both Smiteworks and community) are looking at options - a lot of these are in the conceptual or experimental stage and may change or may not go anywhere. Even with these, the 80/20 rule would be a good level to aim for, with improvements to options and application of effects being made within the rulesets themselves. Some of this I'm already developing in the PFRPG2 ruleset, and some of this is already available in the SWADE ruleset. With the stuff I'm working on I'm very conscious of runtime performance - PFRPG2 PCs are a lot more complex than SWADE ones, so I don't think I can use a similar approach. Which is why I'm looking at things like the experimental features I liked in a previous post. Arcane? - probably, but this is the most efficient I've thought of (for now) and maybe if it takes off I'll look into providing a more user friendly interface to create the base automation code...

It's good to have a discussion, and there will always be widely different expectations and desires across the board. That may result in some people being disappointed, and that's unavoidable. RPG systems continue to evolve and trying to keep a high level of automation with moving goalposts is impossible for a small company that has limited developer resources and that relies on many community developers. I think it's doing a great job so far. Is there room for improvement? Yes, there is - and many devs are looking at doing this. But all of us should keep that 80/20 rule in mind - because we're never going to get to 100%, not even 95% IMO...

[Zacchaeus]

An item of Mithril Plate armor, or really any other variant-type item is simply reskinned by the parsing process, so you get Plate armor that's called Mithril Plate, and a note that it shouldn't cause disadvantage on stealth checks. In order for a different, "correct" Mithril Plate item to be included in a module, there would have to be a separate, Plate armor base item that had that correct property built into it. I guess Mithril plate isn't in the SRD so it can't be the "base item"? I'm still confused by this, but I take Zacchaeus at face value that this is the way it must be.

The images below might help in explaining the idea of templates. Although I'm using the Forge this illustrates exactly how magic items are par5ed.

In the first image the left hand item is the magic item template. As you can see all it has is some text, a value and the rarity as well as some information on the type of armour that the template can be applied to. Next to it is the standard Plate armour from the PHB. You can see that it has all of the properties of the armour according to the table of Amours in the PHB. At the bottom is the combination of the Template and the Plate Armour item and as you can see it has the combined properties of both the template and the mundane item.

The second image is basically the same except that I have created a new Plate Armour called Plate (M), which strips out the disadvantage on stealth and the strength requirement and then combined this with the same magic item template.

So, Mithral armour can be made with the correct properties either in par5e on within FG but it requires the creation of a new mundane armour type in order to do so.

[esmdev]

As I am the person who started the initial conversation about the Mithral Chain Armor in the bug forum, I'd like to add my thoughts.

My current position on the subject is that modules should be C L E A R in the pre-sale and in the module itself that some magical items will require the DM to make adjustments and that some magical items will require active player/DM intervention during play. Some class, race and background features also require intervention to make work without using modules from a third-party website.

While you know, and I know, and most people who've been on the forums for awhile know, that FG isn't a fully automated one-stop does everything new people, or people who don't do forums, might find this difficult to fathom (as some of my players turning DMs that don't read the forums often have problems with).

My original point with the Mithral Chain Armor was that it is part of the module which is released with the D&D Essentials Kit and that the D&D Essentials Kit is aimed at the beginner to D&D. Since it is aimed at the beginner my thinking was that everything possible within the module should function as it is intended in the module. Obviously with the current system this is hard at best and in some cases totally impossible. Still, my feeling was whatever could be done should be done.

Once I decided that the system wasn't really going to work to accomplish that objective, I suggested that something be put in place within the module itself to explain the situation, the limitations, and how to correct them. Maybe even an early story listing of the magical items contained within the module and which function correctly, which need to be edited, and which will never work. That gives the DM an understanding of what is going on out the gate.

One solid change would be...

In the DMG you say: * Optional Game Rules are described but not implemented within Fantasy Grounds. They may require manual implementation
It should say: Some class features, spells, magic items and all optional game rules are described but not implemented within Fantasy Grounds. They may require manual implementation

This should also be added to all of the for-sale systems, modules and supplements to which it is applicable.

[Neovirtus]

That all is VERY reasonable, especially the 80/20 logic. And the fact that not everyone wants/needs that level of automation.

Regarding the "Smiteworks is a small company" line of conversation... it's just strange to me. I would like to point out that many of these effects have already been coded. Third party people sell effects packages for spells, magic items, etc. for associated official modules... which is great I guess? I could go buy them if it were worth it to me. But I do take issue with the idea that it's an unreasonable burden to want (much less expect) Smiteworks to provide the integration for a paid module they produce to work with the software they develop? Especially when someone else goes and sells that integration for $5 on a third party site? I know that's a gross simplification of the issue, but that's the way it looks from the outside.

In the end I get the effort vs. reward logic from a development standpoint. FG is a great product and I will happily continue to use it in its current form and further evolutions. I have no doubt that the development team works very hard and many community developers contribute a lot to the functionality of the software and I do appreciate that. I've personally paid to have the privilege to beta test FGU; I believe in the product. But do I wish that 80% included a bit more? Absolutely.

[Zacchaeus]

That all is VERY reasonable, especially the 80/20 logic. And the fact that not everyone wants/needs that level of automation.

Regarding the "Smiteworks is a small company" line of conversation... it's just strange to me. I would like to point out that many of these effects have already been coded. Third party people sell effects packages for spells, magic items, etc. for associated official modules... which is great I guess? I could go buy them if it were worth it to me. But I do take issue with the idea that it's an unreasonable burden to want (much less expect) Smiteworks to provide the integration for a paid module they produce to work with the software they develop? Especially when someone else goes and sells that integration for $5 on a third party site? I know that's a gross simplification of the issue, but that's the way it looks from the outside.

In the end I get the effort vs. reward logic from a development standpoint. FG is a great product and I will happily continue to use it in its current form and further evolutions. I have no doubt that the development team works very hard and many community developers contribute a lot to the functionality of the software and I do appreciate that. I've personally paid to have the privilege to beta test FGU; I believe in the product. But do I wish that 80% included a bit more? Absolutely.

The third party packages aren't 'coded' as such. The effects are created in a PC actions tab and then dragged to the spells window and exported. They exist only in a module; not within the ruleset itself.

From the posts by Celestian, Trenloe and Mood Wizard (who know more about this than anyone else) including all of that stuff in the ruleset would bog it down to a crawl. At least as the ruleset is currently designed. As Celestian pointed out he had to re-write large parts of his code in order to avoid this; and Trenloe is taking a different approach as well. Both of those authors have in a way the luxury of having a new ruleset to work on rather than an existing one (although the 2e ruleset is based on 5e). So, again, it kinda comes down to development time (it seems to me at any rate). Moon Wizard has been extremely busy with Unity for several years now, and if that hadn't been the case perhaps the 5e ruleset would have been re-written to include more stuff.

Moon Wizard also makes a very valid point as well. A major re-write of the ruleset impacts on the DLC; and that is done mainly by community developers who may not have the time (or possibly the inclination) to re-write the modules.

[LordEntrails]

A possible solution on the mithril armor issue itself is not to create a new template item, but to allow the forge to merge negative properties from the secondary objects. So that when you add Mithral to ppl late armor, the Mithral item has a negative value for stealth disadvantage and therefore removes stealth Dis from the plate template.

Of course, this would require a re write of the forge and the content...

Edit: wishlist item
https://fg2app.idea.informer.com/proj/fg2app?ia=130918

[Neovirtus]

The third party packages aren't 'coded' as such. The effects are created in a PC actions tab and then dragged to the spells window and exported. They exist only in a module; not within the ruleset itself.

So the problem is the method of distribution? Effects can be distributed in "player" modules but not in Official Modules? Or in Spells but not Items? I don't get it.

I think I'm going to stop beating this dead horse and just summarize my position and move on. This started with the Mithril Armor issue, simply as: The item in the module doesn't work as it should, because of the way the coding has to work for distribution of the module (roughly summarized). Fine. My thought was, that shouldn't be the case. And at the same time while we're making items work as intended directly from the module; It would be nice if a simple magical item, with a purely mechanical effect, would have that effect built into it in some way, or at least provided to the player. This could take a number of forms...

All the way from:
1)Full ruleset integration. You equip the item, it applies the necessary effect. Sounds impossible. Ok, moving on.
2)Halfway integration. The item doesn't apply the effect, but it's provided. Either in the item description or elsewhere. Third party software seems to do this in one form or another, seems more reasonable.
3)No integration. The module provides the text of what the item does, but it does not interact with the software in any way.

Currently we are all over this map. Armor +1 is at #1. Ring of protection is at #3. I'd like to see the standard be at least #2, in whatever way it's possible.

Thanks for all the great discussion on the topic, hopefully we get more people's thoughts on the issue over time.

[damned]

I would like there to be some way for FG to indicate to the GM/user - even if its a check mark or text line - that tells us that the particular item/spell/rule needs to be handled manually.

[Zacchaeus]

So the problem is the method of distribution? Effects can be distributed in "player" modules but not in Official Modules? Or in Spells but not Items? I don't get it.

No, Rob2e's packages are created modules. In order to include every effect in the ruleset would require extensive work on the ruleset and as noted above slow it down to a crawl. Effects are provided by the ruleset in order to allow the user to do what the Rob2e packages do. All he's doing is using the ruleset to create the effects, and catering for people who have no interest in creating the effects themselves.

[dulux-oz]

Late to this thread (sorry - that's what I get for being asleep when everyone(?) else is awake) :)

Serious questions to Moon and my fellow Devs: there seems to be two constraints on "automation" - either complexity of (parse) data, leading to longer load times, etc, or complexity of data-structure, leading to larger "databases" (if you're generous with calling a flat-file a database) and thus longer load times, etc.

From a CompSci POV, the solutions to each of these problems (when constrained by "reasonable" commercial limits) is to reduce the size of your flat-file-DB, hence a lot of operations moving away from an XML-based flat-file to something like JSON or YAML; or moving the flat-file to a true DB-System (like MariaDB/MySQL, etc).

OK, so, what are the downs-de of each of these approaches? Well, XML-to-JSON (or YAML) converters are relatively easy to write, so doing the conversions moving forward is (relativitly) not too difficult, and during the transition it wouldn't be too hard for FGU (with its larger memory store) to be able to read both formats. Trouble is, it still means you have to load *ALL* of the data into memory.

The other approach (using a "proper" DB) means more complexity in set-up (you need to include the DB-Engine), but arguments about "more tech-support calls", while once valid a few years ago, have generally disappeared with the modern iteration of "in-app" or "on-CD" versions of modern DB-Engines. The benefit of using a proper DB is that, while the base memory load is slightkly higher, you're not loading all of the data at once, hence the system should run faster overall. The downside, of course, is that its a lot more work to get an XML-DB translater (although it can and has been done). And again, during transition FGU could use both methods.

In fact, if its done correctly, gently, and with careful planning, the end users would never know the difference - only the FG Coders and the Community Devs.

So, a (potentially) radical set of ideas; what do SW and my fellow Devs think - discuss. :)

Cheers

[Trenloe]

I would like there to be some way for FG to indicate to the GM/user - even if its a check mark or text line - that tells us that the particular item/spell/rule needs to be handled manually.
Or maybe the other way around - indicating that automation *will* be applied?

https://www.fantasygrounds.com/forums/attachment.php?attachmentid=31157

[damned]

Or maybe the other way around - indicating that automation *will* be applied?

https://www.fantasygrounds.com/forums/attachment.php?attachmentid=31157

Perfectly acceptable in my opinion.

[celestian]

Late to this thread (sorry - that's what I get for being asleep when everyone(?) else is awake) :)

Serious questions to Moon and my fellow Devs: there seems to be two constraints on "automation" - either complexity of (parse) data, leading to longer load times, etc, or complexity of data-structure, leading to larger "databases" (if you're generous with calling a flat-file a database) and thus longer load times, etc.

From a CompSci POV, the solutions to each of these problems (when constrained by "reasonable" commercial limits) is to reduce the size of your flat-file-DB, hence a lot of operations moving away from an XML-based flat-file to something like JSON or YAML; or moving the flat-file to a true DB-System (like MariaDB/MySQL, etc).

OK, so, what are the downs-de of each of these approaches? Well, XML-to-JSON (or YAML) converters are relatively easy to write, so doing the conversions moving forward is (relativitly) not too difficult, and during the transition it wouldn't be too hard for FGU (with its larger memory store) to be able to read both formats. Trouble is, it still means you have to load *ALL* of the data into memory.


Moving from XML to JSON wouldn't be a bad idea but XML is also how the UI is laid out. Trying to edit JSON for UI/controls would be a nightmare... and that's saying a lot (I don't like XML either). HTML templating for UI control would be a viable option (much more common).

I'd be curious to see how XML to JSON would improve things. My understanding of the resources/speed issue is tied to XML "nodes". Each one is a LUA object and because of that it requires more resources. JPG has suggested they could be un-tethered but it would require a major re-write of code which... isn't on the table.

[esmdev]

I like the automation A. It looks like a great solution and it's easy to see.

[Neovirtus]

Agreed, I think that makes a lot of sense. Let people know what is fully automated, and they will know to double check everything else.

[deer_buster]

Late to this thread (sorry - that's what I get for being asleep when everyone(?) else is awake) :)

Serious questions to Moon and my fellow Devs: there seems to be two constraints on "automation" - either complexity of (parse) data, leading to longer load times, etc, or complexity of data-structure, leading to larger "databases" (if you're generous with calling a flat-file a database) and thus longer load times, etc.

From a CompSci POV, the solutions to each of these problems (when constrained by "reasonable" commercial limits) is to reduce the size of your flat-file-DB, hence a lot of operations moving away from an XML-based flat-file to something like JSON or YAML; or moving the flat-file to a true DB-System (like MariaDB/MySQL, etc).

OK, so, what are the downs-de of each of these approaches? Well, XML-to-JSON (or YAML) converters are relatively easy to write, so doing the conversions moving forward is (relativitly) not too difficult, and during the transition it wouldn't be too hard for FGU (with its larger memory store) to be able to read both formats. Trouble is, it still means you have to load *ALL* of the data into memory.

The other approach (using a "proper" DB) means more complexity in set-up (you need to include the DB-Engine), but arguments about "more tech-support calls", while once valid a few years ago, have generally disappeared with the modern iteration of "in-app" or "on-CD" versions of modern DB-Engines. The benefit of using a proper DB is that, while the base memory load is slightkly higher, you're not loading all of the data at once, hence the system should run faster overall. The downside, of course, is that its a lot more work to get an XML-DB translater (although it can and has been done). And again, during transition FGU could use both methods.

In fact, if its done correctly, gently, and with careful planning, the end users would never know the difference - only the FG Coders and the Community Devs.

So, a (potentially) radical set of ideas; what do SW and my fellow Devs think - discuss. :)

Cheers

Well, as long as we are speaking hypotheticals here, as someone with a software engineering background, I'd love for there to be a "proper" DB. Hell for that matter, I would love to get rid of LUA altogether and have a "proper" way to extend FG, but I realize that LUA is the common game scripting method currently. XML is outdated and unwieldy, and templates would be my preferred replacement.

[LordEntrails]

I too like the A-Automated idea. Though I would like the A to be more unique/identifiable. Maybe a circle A or a script font A.

[Bidmaron]

A gear icon would be perfect