Massive memory spikes causes PCs being unable to play

[mini1337s]

Setup: Ultimate edition monthly licence. Setup through Steam, DND 5E core books purchased by one player (myself). Players connecting to me through Demo Fantasy Grounds version.

With only the PHB loaded for players, players see their memory usage for the program go up to 1.4 gigs and beyond, which causes the software to lag. Stuttering effect makes the game unplayable for them (doesn't effect my game). Only playable for them when PHB module is removed. Looked for troubleshooting on forums, the general consensus for similar issues is to remove modules, but only the player's handbook is loaded for them. Do they need to purchase their own versions? Can I export this information? Is there a workaround? I'm a bit over $200 deep now so this is disappointing.

Any advice as to the next step?

[Trenloe]

Welcome mini1337s, certainly the PHB takes a lot of memory (usually around 800MB). How much memory was FG using before the players opened the PHB module? In a quick check I've just done mine was 200MB and jumped to 1,000MB when I opened the PHB. Do you have a lot of tokens in the \tokens\shared directory, or any other assets (lots of maps) manually shared?

Do you have any of the other basic rules or SRD modules available for the players to open (a green check/tick in the Module Activation screen)?

Additionally, what type of operating system are the players using? If they are using a 32-bit operating system then they'll definitely be struggling - FG is limited to under 2GB of memory use in a 32-bit operating system, and you start to see issues around 1.5-1.6GB of memory use. 64-bit operating systems work a bit better, but are limited to around 3.6-3.7GB of FG memory use.

[Nylanfs]

Also have everybody turn off the steam overlay.

[mini1337s]

Hello Trenloe,
All of us at on Windows 7, 64 bit. The memory usage sits around what you are seeing, 200mbs then jumps with PHB, but jumps to that 1 gig mark then continues to climb. Within 30 seconds, they are at the 1.4/5 gig mark and it stutters too much to play.
I've tried clean games, literally just create a campaign and load the PHB. I've tried removing the PHB tokens directory but that also removes PHB in its entirety. Nothing is being shared to my knowledge, just the info in the PHB guide.

EDIT: At Nylanfs
We just tried this per your suggestion but the issue persists.

[Trenloe]

If you want me to try from my side and try to see what's happening then please PM me either your FG connection details (Alias or GM IP address) and I'll connect as a player and try to help.

[mini1337s]

Thank you Trenloe. I've PM'd you the IP address.

[mini1337s]

After some troubleshooting, it appears that the issue was related to the first time loading of the PHB info. A combination of the current software, increase in memory usage, and internet bandwidth seemed to cause the issue.
If you are looking at this in the future and have a similar issue, try waiting for 10-15 minutes with your players to see if that loads all the info and fixes the lag.

[Nickademus]

An alternative that will soon be possible, is (when version 3.2.2 goes live) export all the assets from the PHB into a module of its own and use that. It won't have all the images that cause the PHB's massive memory use.

[Talyn]

I write my own Reference Manuals in XML with images and don't notice crazy memory increases. I have to wonder though if the PHB was constructed with PAR5E because PAR5E creates the search keywords in the most inefficient way possible: it duplicates the entire body of text word for word, including all duplicate words, which effectively doubles the file size and every page you read loads those keywords into memory to be parsed through in case the user does a search.

Could be the images though... I'll have to look at memory consumption with my refmanuals vs. the PHB. Zacc is really good about setting correct image quality to decrease file/memory size though.

[Nickademus]

Interesting. Though what I've heard is that it's the why modules are handled going from the host to the client. The host won't notice a problem. (I've ran FG using 800 MB with the PHB, but it uses ~3 GB as a player). So you'll need to test your manuals with other players.

[damned]

I write my own Reference Manuals in XML with images and don't notice crazy memory increases. I have to wonder though if the PHB was constructed with PAR5E because PAR5E creates the search keywords in the most inefficient way possible: it duplicates the entire body of text word for word, including all duplicate words, which effectively doubles the file size and every page you read loads those keywords into memory to be parsed through in case the user does a search.

Could be the images though... I'll have to look at memory consumption with my refmanuals vs. the PHB. Zacc is really good about setting correct image quality to decrease file/memory size though.

Of course the PHB is done with Par5e. That was what Par5e was written for!

[Nickademus]

... because PAR5E creates the search keywords in the most inefficient way possible: it duplicates the entire body of text word for word, including all duplicate words, which effectively doubles the file size and every page you read loads those keywords into memory to be parsed through in case the user does a search.

I had noticed this too. Though I wonder, does FG load the keywords into memory when the module is loaded or when the reference manual window is created for display.

[Moon Wizard]

All of the database data and token data is loaded into memory when the module is loaded, except images on the GM side, which are delay loaded.

Regards,
JPG

[Nickademus]

So how difficult would it be to purge the keyword data from the PAR5E-made modules? How difficult would it be to replace it with actual keywords?

[Bidmaron]

Now that we have a Pathfinder license, when will PF fans see a PARPF?

[Talyn]

Unless something changed with 3.2.1 I noticed keywords don't get loaded (search will fail) until you actually read the page (click it in the index to render the content). If we use any PNG images (specifically PNG using the "token" call in order to render the alpha channel) in the manual those remain in memory the whole time.

[Nickademus]

Now that we have a Pathfinder license, when will PF fans see a PARPF?

Never. PAR5E is third-party software not made by Smiteworks. If no one has made a parsing program for PF content yet, it probably won't happen.

That said, some of the people making the Store PF content have their own parsers that pull from the PFSRD (such as Trenloe's bestiaries and spell modules). But these haven't been made public since the end result modules were made public.

[Nickademus]

Unless something changed with 3.2.1 I noticed keywords don't get loaded (search will fail) until you actually read the page (click it in the index to render the content). If we use any PNG images (specifically PNG using the "token" call in order to render the alpha channel) in the manual those remain in memory the whole time.

Well, I can tell you that (from speaking with the designer) the PHB doesn't use any tokens in the reference manual. It only uses embedded images. (A.k.a. all <image>, no <picture>.) So that wouldn't have an effect on the PHB using memory.

[Talyn]

[Del]

[damned]

Well, I can tell you that (from speaking with the designer) the PHB doesn't use any tokens in the reference manual. It only uses embedded images. (A.k.a. all <image>, no <picture>.) So that wouldn't have an effect on the PHB using memory.

Embedded images are stored as tokens in Par5e generated material :)

I imagine Par5e is also being used for Pathfinder Reference Materials and has probably been extended to do so. The team that do the D&D stuff are very familiar with it and it would make sense to use familiar tools to do new content.

In regards to maually stripping and replacing the keywords outside of Par5e the issue would probably be that new versions of some of the sourcebooks are possibly created almost daily as bug reports come in. A manual update would be tedious indeed....

[Moon Wizard]

The reference manuals actually have an option for both token type images as well as standard images, so it depends on how it was built.

Regards,
JPG

[Nickademus]

So does PAR5E change the image element?

[Nickademus]

I imagine Par5e is also being used for Pathfinder Reference Materials and has probably been extended to do so. The team that do the D&D stuff are very familiar with it and it would make sense to use familiar tools to do new content.

I would think that the people that already made parsers for PF would be using those instead of PAR5E. I'm pretty sure that's why Trenloe was asked to do the bestiaries (because he already had a parser for them). But that is just speculation.

[Talyn]

In regards to maually stripping and replacing the keywords outside of Par5e the issue would probably be that new versions of some of the sourcebooks are possibly created almost daily as bug reports come in. A manual update would be tedious indeed....

It's tedious no matter what. I really enjoy constructing the reference manuals but I absolutely dread and loathe every moment of generating the keywords.

[damned]

I would think that the people that already made parsers for PF would be using those instead of PAR5E. I'm pretty sure that's why Trenloe was asked to do the bestiaries (because he already had a parser for them). But that is just speculation.

For sure. By reference I did mean reference manual.

[Nickademus]

I see. Then I assume that the team you are referring to are the ones that have the professional version of PAR5E (since no one else can use it for reference manuals). I don't keep track of who all they are and honestly forget sometimes that PAR5E can do ref manuals since we can't use it to make them.

In context, that would make sense, though I'd like to see more people involved in quickly converting the PF material than just those with Pro-PAR5E.

[damned]

I see. Then I assume that the team you are referring to are the ones that have the professional version of PAR5E (since no one else can use it for reference manuals). I don't keep track of who all they are and honestly forget sometimes that PAR5E can do ref manuals since we can't use it to make them.

In context, that would make sense, though I'd like to see more people involved in quickly converting the PF material than just those with Pro-PAR5E.

Ive written reference manuals with the public Par5e - you do have to do some manual editing post par5e. Im referring to people like Zacchaeus, Zeph and others who did the 5e content. As you and many others do you can write reference manuals manually. Additionally there is Ikaels new tool if you own the Savage Worlds ruleset - you can edit the ruleset tags to other systems manually at the end.

My comment was an observation/guess.