Complete OSRIC mod preview for feedback

[First Edition Society]

Edit - please see this thread (https://www.fantasygrounds.com/forums/showthread.php?70079-Authorized-OSRIC-mods-for-preview) with the up-to-date attachments, and discussion/comment for the First Edition Society OSRIC mod.



Hi all,

Works been ongoing for a while on an OSRIC mod designed to work with the official AD&D 2E Fantasy Grounds rulebook and Celestian's 1E extension. The mission of this mod is to automate many of the processes in the OSRIC book, and provide all the widgets necessary to create content and run campaigns using Fantasy Grounds. This is intended to be the platform for additional work by the First Edition Society on Fantasy Grounds for more OSRIC content.

It's a big mod. Work in FGC started in March, and the mod is butting up against FGC's 32-bit limits. The mod itself is complete and I've verified the processes all work, but the ref manual can't be finished. I'll need to break it up probably in order to make it more modular so DMs can only load the portions they'd use in running a live game and not the portions they'd need when creating content for the game to be played.

If you have a moment, I'd appreciate comment/feedback: any table glitches, points of division for separating into multiple mods, etc. This is meant to go up on the forge and will be free. Try out the generators - NPC parties, caravans, pilgrims - all the stuff that is the tip of a gigantic process tree iceberg mostly underwater. Hit the "roll" button and go get yourself a frosty beverage. It might be done by the time you get back ;)

Suggest running it in FGU, as it's scraping the ceiling of FGC.

I'll be AFC for a few days, but will respond to any and all feedback as I can. I appreciate the communities pointers for how to optimize this for Forge distribution!

[Moon Wizard]

If it's hitting FGC limits on a single mod; then you probably need to look at shrinking the graphics to more manageable sizes anyway for both load times and transfer speeds.

Regards,
JPG

[celestian]

If it's hitting FGC limits on a single mod; then you probably need to look at shrinking the graphics to more manageable sizes anyway for both load times and transfer speeds.

Regards,
JPG

The OSRIC book is effectively the PHB/DMG/MM all in one.

I think he plans to split out some of that but right now the OSRIC Full is everything and at 9.2 meg. Which, these days, is a microscopic download in the scheme of things.

[EOTB]

If it's hitting FGC limits on a single mod; then you probably need to look at shrinking the graphics to more manageable sizes anyway for both load times and transfer speeds.

Regards,
JPG

There's hardly any graphics in it. It's just the number of game items. The graphics that are in it were put through photoshop to get them down to a few kb. I'm trying to think of separations between what people would need when playing, vs what people would need to prep and export content those tables create vs being loaded at the same time as an RPG session is ongoing and never used in the game session itself.

[Trenloe]

What isn't microscopic is the db.xml file in the main module - 81MB uncompressed, and over 2 million lines of XML (2,353,694 lines to be precise). This is the largest data module I've seen by a long shot - and no surprise that FGC struggles with it and runs out of memory.

@First Edition Society - FGU takes longer to process data than FGC did. Just opening the full module took around 2 minutes on my computer - which is a pretty high end i9 based Windows computer. The memory use went from 340MB to 7,500MB.

Monumental task putting this together! But it's a little too monumental!!! The main areas that have a lot of records are Table (over 3,700 of them) and NPCs (over 2,000).

So, definitely needs some trimming down and segmentation into individual modules to allow control over what gets loaded.

[EOTB]

Yeah, I think I’ll have to give the final form a break up.

There was no degradation in performance until I put the thin mint of the ref manual on top. Then it puked everywhere

[JohnD]

This is a huge bit of work... I also suggest breaking it up into smaller parts.

Love the ability to generate complete adventuring parties.

[EOTB]

Thanks John!

While AFC, I’ve been thinking about ways to split it between use-to-play and use-to-create.

Play - always load: the players reference (already split) as mod 1 and the monsters chapter as mod 2. For people who mainly play pre-packaged scenarios, this is all they’ll need to load for most adventures on a “dungeon” map such as G1, U1, B2, etc.

Play - sometimes load: dungeon random encounter tables (mod 3); wilderness random encounter tables (mod 4); urban random encounter tables (mod 5). Either mod 3 or mod 4 would likely be loaded with mod 1 and 2. But that three-mod load should be the most you’d need at one time in-game with players, and I think would be a similar computer tax as the 2E computer tax. Mod 5 would be expanded with the handful of monsters on the urban encounter table so the DM wouldn’t have to use mod 2 at all, with its hundreds of not-applicable monsters to urban adventuring. But you’d need to load one of the content creation mods below (mod 6 - NPC class templates). The urban tables pull a surprisingly high number of PC classes of various races and levels into that encounter table. So there’s no benefit to surgical redundancy as with the monsters. That results in urban adventures also having a 3-mod load

Content creation: NPC class templates (mod 6, often but not always used with mod 7); treasure tables including monster lair treasure generation (mod 7); NPC adventuring party generation tables (mod 8, loaded with mod 6 and 7 when used in prep); and random dungeon generation tables (mod 9).

Module 1 nearly always must be loaded in prep as well, of course.

NPC adventuring party tables represent a huge load (400+ tables and other widgets) that would only be drag when not generating NPC parties.

I’m wondering if the NPC templates wouldn’t be better as read-only so that anyone new to FG using the OSRIC mod(s) doesn’t use the template to make a specific NPC, overwriting it in the process.

It seems to me that I can load exported adventures (e.g., N1) with treasure parcels and the magic items embedded in the adventure don’t require the 2E DMG to be loaded to work. Am I remembering correctly? If so, the treasure tables can largely be shunted to prep tables instead of play tables, presuming most GMs make their parcels outside of play.

All of these seem to me to be tables I would use before logging in with five buddies to run a game. And if I am doing campaign-style overland travel where players have a random encounter designated “in-lair”, where they’ve stumbled onto an unplanned monster having some sort of treasure hoard, the GM can load the treasure module after the fight, generate that single treasure with one click while the players take a quick break and high-five each other, and unload the treasure module afterwards.

Does that look like a usable segmentation? Or is that now *too* segmented?

It would take a little while for me to rebuild to that plan, as I think I’d need drag-export the applicable items in batches into their new mods, open up a new campaign folder for each mod, open the exported mod into its new home campaign folder and then drag copy each item so I have a clean folder for fixes and re-exports into updated mods as-needed.

Plus I’d have to “re-drop” all the shields as I go so that where mods do interlink (e.g., monster treasure buttons in a monster description), it’s telling the DM to load the new segmented mod and not “full OSRIC” to use that button.

But that process will still be much faster than it was to make the various items from scratch; it’s doable.

[EOTB]

Also wondering: should I make the DM’s half of reference manual it’s own mod so they can always have the full ref manual at their fingertips even if not loading some mods.

I don’t think (?) a reference manual by itself takes up much space/resources if cosmetic JPGs are left out.. It seems more convenient than having many mini reference manuals loading with segmented content mods.

Feedback here would be appreciated also.

[Trenloe]

Other ruleset modules do full reference manuals in one module. The Pathfinder Second Edition Core Rules module, for example, has full reference manual of the base 640 page rulebook/PDF, and this includes a lot of images too (200 in total). The reference manual in itself is not an issue - it's the shear number of records and amount of data that becomes an issue.

As I mentioned earlier, the full OSRIC module linked in post #1 contains 2.35 million lines of XML. The PF2 core rules I've just mentioned, which is a large module by comparison to most, weighs in at "just" 75,000 lines of XML (30 times smaller than the full OSRIC module). Now, just counting lines in an XML file doesn't give a complete picture, as some lines may have a lot of data (especially those with text) whereas others may just have a simple number, but it gives a good idea of the shear huge size of the module.

You can also open up the db.xml file in an XML capable editor such as Notepad++ and look at where the biggest type of data is being used. Note - this will take a long time for such a big file, and Notepad++ will go "not responding" a few times - just leave it to complete what it's doing.

Here's the db.xml file from the full module, with the first level of records collapsed - noted the line numbers to the left of the main FG record types:

https://www.fantasygrounds.com/forums/attachment.php?attachmentid=48493

Looking at the big records:

NPCs - 1,750 thousand lines
Tables - 331 thousand lines
Items - 150 thousand lines
Reference - 55 thousand lines (this primarily includes the Reference Manual).
Classes - 30 thousand lines
Spells - 25 thousand lines


So, this ties in with what I mentioned earlier - NPCs and tables are the main areas, with NPCs being massive compared to the rest - taking up 75% of the whole module!

If I just remove the NPC data from the module, in game module load time drops to around 20 seconds and memory use is at about 1.7 GB (compared with over two minutes and 8 GB for the full module).

So, I'd recommend that you split this into at least three modules - maybe more.
1) A player based module - classes, items related to players, spells, just the relevant tables, etc.. This will still be a big module, but can be loaded by the GM and players and won't overload the player side. Player relevant reference manual.
2) A GM based module (not including NPCs) - GM related tables, magic items, treasure parcels, etc.. GM relevant reference manual.
3) At least one NPC module - I've recommend splitting these further along the lines of the categories/groups you have - allowing the GM to open the relevant NPC module/s as needed, and close them down after. The following screenshot shows where the most data is in each NPC group/category, this will give you an idea of where to split your NPC modules up (for example, Demi-Humans has 570 thousand lines):

https://www.fantasygrounds.com/forums/attachment.php?attachmentid=48495

[EOTB]

Thank you Trenloe, that is very helpful. I draw a few conclusions from it:

1) In best practices, it talks about crosslinking but doing so isn’t “free”. All the heaviest NPC groups are heavily cross-linked. I would bet that the players half of the reference manual, which is the only completed half, has many more XML lines than the mostly unlinked, unfinished GMs half.

Cross linking is still a best practice, it’s just that advice considers smaller data groups than something like the entire 1E core rulebook set heavily cross-linked all over the place.

For example, many of the demihuman templates in OSRIC/1E contain the thieves skills. All assassins contain those plus another dozen assassin skills. Gnome and dwarf assassins contain all of the above plus another half-dozen racial skills for determining depth underground, etc.

Then every Demi-human will have at least three cross-links on its description page to NPC magic item determination tables relevant to their level, so the GM can just push a button and get personalized magic items, and other similar tables.

Pilgrims, caravans, and NPC adventuring parties all together weigh about as much as demihumans even though they’re about half as many combined entries. These, as completed characters would have fully complete spellcasters and assigned magic items. It wouldn’t be uncommon for a magic item to have a half-dozen crosslinks to spells, per item, and of course a mid-level spell caster could easily have 20-30 spell cross links in “spells known” in addition to voluminous magic items. A staff of power probably has a couple dozen cross links or more by itself

Takeaway - cross-linking is a valuable time convenience but is not the negligible additional weight I had thought.

Since pilgrim, caravan, and adventuring party NPCs were added immediately before I tried to finish the ref manual just before uploading, that 400K+ additional lines of XML was like adding 5 full starcraft mods to the mod. And thus it redlined.

But one of the objections I’m trying to meet with the module are those who say using a VTT means twice the work: first to make the content as they would without a VTT, and then the work of remaking the content in the VTT. I think I’ve succeeded at actually making it faster to generate campaign content directly in the VTT. I will have to figure out the resource load.

You’re right: NPCs are the issue. And I have to manage so a GM never has to load everything at once.

But I don’t think I can keep everything else on the DM side together and just split the NPCs, because the encounter tables pull across many of those NPC groups at the same time. The DM would have to load dragons, giants, humanoids, animals, sylvan, undead, men, other monsters, etc., simultaneously for the wilderness encounter charts to work. They’d need to load a subset (10%) of the current demihuman group as well.

But I think I have a path. I won’t need to segment it quite as much as I said above, but probably still 6-7 mods with the segmenting along functional lines instead of type lines. If I segment by type the DM still has to load nearly all of it sometimes.

But I need to cut out all redundant cross linking. Example: the base orc entry has several cross links. Those are repeated again in the orc sub-chief entry, the orc shaman entries, and witch doctor entries, etc., because all of those are replicated/tweaked versions of the base orc entry.

Repeat that redundancy across a lot of monsters and that’s probably a Starcraft mod or two by itself in excisable bloat. I only need the cross links in the base monster entry.

[EOTB]

One other question: is it a less expensive XML cost to call upon a table by copying/pasting the table name between brackets than it is to shield-drop the same table, when making it a sub-table in the cell of a larger table?

Right now I mostly have shield-drops as subtables since it allows clicking through to the sub-table. But if that creates more lines of XML than bracketing the name then I may not be able to afford the convenience

[celestian]

One other question: is it a less expensive XML cost to call upon a table by copying/pasting the table name between brackets than it is to shield-drop the same table, when making it a sub-table in the cell of a larger table?

Right now I mostly have shield-drops as subtables since it allows clicking through to the sub-table. But if that creates more lines of XML than bracketing the name then I may not be able to afford the convenience

I think the benefit to typing the name is it wont require a a direct link to access it... it will look for a table with that name. At least thats how I understand it. Using direct link will mean if you change the name of the module/rebuild and the ID changes, it'll break.

[EOTB]

Edit - I misunderstood. You mean if bracketing the text of a name, rebuilding it doesn’t “break” the sub-table reference if that table moves from “OSRIC full” to “OSRIC module #4”?

Whereas with shield dropping, it would break the master table in that circumstance(?)

[readymeal]

Hi EOTB,

i have tried your mod as discussed over the weekend, i loaded the GM part and the players only loaded their part...

Sure the initial loading time was longer than usual but the subsequent ones were a lot shorter and mildly longer than opening any other 2E campaign. I don t remember any of the players complaining about loading the players part...

It works flawlessly to create characters. just had to remind them to drag the skills associated to their race and class if needed. Also noticed that Character creation needs to be in certain order... putting characteristic, race then class does not work well...

Finally well yes it s a big mod, it contains a lot of stuff... now we should maybe stop expecting people making mod built in 2021 to load perfectly in FGC (RIP) or FGU... I am sure you can tweak your mod... as can FGU be optimised by the dev. It goes both ways.

IMO no one for example who played the original Battlefield 1942 when i came out is expecting to play the latest instalment with the same rig...but the FG community think so...well they are somehow right since not much changed practically beside LOS and lighting and a map making tool with very...very limited usage. I see FGU as having untapped potential being held back by his predecessor. Anyway i diverge here...

Honestly EOTB, you have done a lot of work to get this out and i know there s more in the pipeline, I would not bother cutting it off in pieces. It s working fine, only the initial loading is longer... oh well, i can live with that.

[EOTB]

Thank you for taking it for a spin Readymeal, I appreciate it! Glad to hear you ran a game without hiccups.

I’d be interested in hearing about character creation, as stats-race-class in sequence was how I was creating them, so if you found any glitches doing the same I should track that down.

Also good feedback on the splitting issue. I need to weigh that with the size of the work in the pipeline, as if the dangerous dungeons supplement to OSRIC is released on FG, that’s about as much new content as is in OSRIC itself...

[Trenloe]

Whatever you decide to do, you should consider the minimum system requirements stated for Fantasy Grounds Unity here: https://fantasygroundsunity.atlassian.net/wiki/spaces/FGCP/pages/1136984087/System+Requirements

At present, the OSRIC full module makes FGU use nearly 8GB of RAM just opening it - that's going to take twice the minimum spec RAM of 4GB (even before the operating system uses any); let alone playing on a map, with a few players joined and a bunch of monsters in the combat tracker. People on the recommended hardware (8GB RAM) are still going to struggle. So my recommendation would be to provide some way for people on FGU minimum spec machines to be able to fine tune the amount of memory used by having different modules to load/unload as needed.

[EOTB]

I expect to split it, over a period of time, because there’s more work in varying stages of completion that builds on this - more monsters, more races/classes, more spells, etc.. So loading “everything OSRIC-compatible” at once (speaking in general and not this base mod specifically) won’t likely be possible anyway, even for those with premium machines.

But I’ll probably keep an all-in-one GM’s half available for those who want it - maybe vanilla OSRIC is all they plan to run and their system can handle it.

[celestian]

I expect to split it, over a period of time, because there’s more work in varying stages of completion that builds on this - more monsters, more races/classes, more spells, etc.. So loading “everything OSRIC-compatible” at once (speaking in general and not this base mod specifically) won’t likely be possible anyway, even for those with premium machines.

But I’ll probably keep an all-in-one GM’s half available for those who want it - maybe vanilla OSRIC is all they plan to run and their system can handle it.

To be clear, it loads fine for me in FGU tho it does take about a minute+ to load.

If it was me I'd break out the DM only into it's own module excluding monsters. Make a module specifically for monsters. Then player version with all the class/race/spells/normal items and non-DM specific instructions.

[EOTB]

That would be pretty simple to do

[readymeal]

To be clear, it loads fine for me in FGU tho it does take about a minute+ to load.

If it was me I'd break out the DM only into it's own module excluding monsters. Make a module specifically for monsters. Then player version with all the class/race/spells/normal items and non-DM specific instructions.

Yeah i think Mike is on the money here,

a simple question i was wondering, is it better for FGU to run 3 smaller modules together or just 1 big one...knowing the total information is the same?

Another way would be to rethink the whole thing... what is needed to create characters etc, what is needed to run an adventure...and finally what is needed to create an adventure...just a thought.

Ultimately, with the additional content coming down the pipe...you need to find a the best way to run this looking at FGU limitations (is it not based on a game engine...)

[Trenloe]

a simple question i was wondering, is it better for FGU to run 3 smaller modules together or just 1 big one...knowing the total information is the same?
Ultimately, the memory use of the data will be the same.

Having the module split will help those who have minimum FGU spec computers. Saying something loads fine for you if you don't have minimum FGU spec isn't really a good indication of how a module will perform. I have an i9 CPU with 64GB RAM and, yes, it works fine for me - but the 7GB memory use (even before I start playing) will be an issue for quite a few users. The 2 minute load time I mentioned in my testing is a one off, so the impact goes away once loaded for the first time - but, again, for some people this will be an issue and will be reported on the forums - as they'll see FGU not responding for two minutes (or however long it takes on their computer) and report this as a "crash" or something similar.

@EOTB - I made recommendations in post #10 about at least a three module approach - in order to target as many users as possible, I think this is the best approach. If there's not much interest in making this usable for all users then it will be good to make it very clear in whatever location/thread/website the module/s made available for download as to the memory use, load times, etc.. People may still not read that or take it on board, but at least you're clear the impact of loading such a massive amount of data.

[EOTB]

As mentioned I expect to segment it for the lower-spec machine users. I just don’t think I can segment it on record type for them and achieve the dual goals of full utility + keeping the load on their machine low.

I would think the 3-mod split will be all loaded at once 90% of the time OSRIC is used in play. (Unless FG GMs routinely unload monster mods during play). The primary benefit to a 3-mod split is prep, where having every monster in the book at-hand isn’t necessary for a lot of prep functions. So the 3-mod split isn’t any sort of solution for lower spec machines, it is a lower-effort convenience for higher-spec machines.

For lower-spec machines I think I’ll have to segment by functionality, where I separate based on what is virtually never used with other parts. If adventuring in a dungeon, ocean and under-the-sea content is a waste of resources. Same for lost world content when running a shire, and bulettes if running a waterborne pirate adventure.

Prep modules broken out separately, etc.

Those who need that sort of thing will load a sort of “mix” of abridged OSRIC containing only material pertaining to the activity at hand, and not loading any that doesn’t. The number of people using that approach may grow as the amount of content pertaining to “the activity at hand” continues to grow - it will likely double, at least, in the next while.

But what I’ve done so far is running a bit behind schedule, so this split for lower-spec machines will have to wait a bit. On the other hand, cutting the GM part in half for those with better machines I can do more quickly and will be priority, so will be done first along with finishing the ref manual to have have a product for the Forge.

I will note the present performance requirements and note that a series of segments/abridgments will be put out for those needing them.