FGU Memory Leak

[BushViper]

I'm making this post against my better judgment, but at the request of one of our moderators.

My problem is that my players and I experience a level of lag/slowdown that makes FGU unplayable after awhile during our sessions. I say "awhile" because the slowdown doesn't seem to follow a pattern. Sometimes it's an hour, sometimes three.

We play 5e, FGU, DM and 5 players. My computer is a monster relative to the system requirements for FGU.

I've been reading and researching about performance issues for some time now, but it's become particularly pressing because two of my players find it so irritating they're wanting me to switch to Foundry. I don't want to do that so I am desperately looking for help, but I don't want band-aids or duct tape; I want the base problem looked in to.

Anyway, the research points to the chat memory leak that others are experiencing. Those that are dismissive of the report that the issue is an extreme corner-case that no 'normal' user will ever encounter are, well, patently wrong.

The extreme die-rolling scenario is simply an easily reproducible example of the problem that will literally crash FG, but what I, and many others, are experiencing isn't as extreme as a crash, but there is a consistent and dramatic slowdown in performance. Tokens stutter and move like they're in molasses, pressing buttons get no or a very laggy response, maps won't zoom in/out, etc., etc.

I think the problem is far more common than people realize, but it's particularly evident for power-users that use a lot of extensions that dump a lot of information in to the chat window that isn't normally there. The extra information provides a similar (though not as extreme) environment to rolling 10k dice.

Anyway, it's real. FGU definitely has a memory leak (or some other, but similar, broken resource management issue) and it needs to be examined.

[Zacchaeus]

Can you post the campaign folder.

[BushViper]

Can you post the campaign folder.

I'm reading about how to do that, but I will as soon as I figure it out.

[Moon Wizard]

The challenge is that while we have heard of some people that are encountering lag after a few hours, we have not been able to get any good information about what is causing the issue. While I would definitely want to make sure that we get your campaign files; I do not have the ability to run all the same extensions and sessions that you do in a meaningful way. (because I don't know their code nor how to use the extensions). Have you tried playing without the extensions and get the same problems?

So, it would be great to get information about the state of the tabletop at the time that the slow down is occurring. (i.e. which windows are specifically open? what was the general flow of play for window usage?)

Also, it would be good to get information about the scenario in progress, in terms of testing a few scenarios.
When the slowdown occurs, can you try a few things:
* Have one player shutdown FGU completely and restart/connect from scratch. Do they see the same slowdown after rejoining, or only the people that didn't reset?
* Have the GM close all IMAGE windows. Does the GM see the same slowdown after re-opening only the current map? Do the players still see the same slowdown after this test?
* Have the players close all IMAGE windows. Do they see the same slow down after re-opening the single map window in progress?
* Have the GM close all windows. Does the GM see the same slowdown after re-opening only the current map and combat tracker? Do the players still see the same slowdown after this test?
* Have the players close all windows. Do they see the same slow down after re-opening the single map window in progress?
* Does having the GM shut down and restart the session fix the slowdown when the players reconnect?

Regards,
JPG

[Moon Wizard]

Also, are you sure it's a memory leak that is causing the slow down? Are you running the Windows Task Manager when this is occurring? What are the numbers when the campaign is first opened? What are the numbers when the slowdown is occurring?

Thanks,
JPG

[BushViper]

I've been aware of the problem for months, but I got completely fed about two sessions ago and started paying a lot more attention to the problem and began trying numerous things to fix/mitigate the problem. One of my players (who is in tech) even said that it behaves like a memory leak. However, I don't have any notes or definitive information I can pass on, yet.

My session isn't until Thursday (12/2), but during that game I'll take meticulous notes and try each of the suggestions in your post. At least then we'll have a better foundation to work from. Though, I am extremely confident it's a memory leak. It behaves like one and gets relief from the same practices that address memory leaks (i.e. completely closing the program).

Also, I spoke with Zacchaeus in Discord earlier and agreed that I need to run a session without my extensions to get a comparative result. Though, I've already conceded the point that I think the problem is likely due to the sheer amount of chat information that the extensions (collectively) produce. The Stealth Tracker extension alone produces a ton of chat data that gets compounded by each entry in the combat tracker.

[Moon Wizard]

I'm definitely not arguing against the fact that it may be a memory leak; but I don't want to assume without data. It could also be a network loop or other ongoing process buildup, not just memory. I just want to get more data, and see if we can't get more detail. (i.e. if closing image windows only resets issue; then we only need to look at images; etc.)

Regards,
JPG

[seansps]

I have the exact same issue you are describing, however, I use less extensions (and don’t use any that dump a lot of chat data like you describe.)

So there is definitely an issue. I will try to gather data as well after my next session and report back if I find anything.

[LordEntrails]

I have the exact same issue you are describing, however, I use less extensions (and don’t use any that dump a lot of chat data like you describe.)

So there is definitely an issue. I will try to gather data as well after my next session and report back if I find anything.
If you use Windows Performance Monitor during your session, it will capture a log of your system during play. I would think capturing the metrics of CPU, RAM, disk I/O, and graphics card/video would be what you want to capture.

[seansps]

If you use Windows Performance Monitor during your session, it will capture a log of your system during play. I would think capturing the metrics of CPU, RAM, disk I/O, and graphics card/video would be what you want to capture.

Agreed. I’m running on MacOS however, so I’ll have to just take screenshots of RAM and CPU utilization from Activity Monitory when I notice it happening. (I’ll gather logs, as well.)

If there is a better tool for MacOS to collect metrics, I’ll use that- I’ll look around for some.

[Lo Zeno]

If there is a better tool for MacOS to collect metrics, I’ll use that- I’ll look around for some.

If you're comfortable enough to run a python script, there's Syrupy:
https://github.com/jeetsukumaran/Syrupy

I use that when I work on macs to log CPU and memory usage of specific processes.

[seansps]

If you're comfortable enough to run a python script, there's Syrupy:
https://github.com/jeetsukumaran/Syrupy

I use that when I work on macs to log CPU and memory usage of specific processes.

Thanks! I’ll try this for sure. As a developer myself, it’s no problem.

[LordEntrails]

Agreed. I’m running on MacOS however, so I’ll have to just take screenshots of RAM and CPU utilization from Activity Monitory when I notice it happening. (I’ll gather logs, as well.)

If there is a better tool for MacOS to collect metrics, I’ll use that- I’ll look around for some.
Looks like there are paid tools. But the link below has a command line that takes Activity Monitor data so many times every so many seconds and builds a log file. Might be easy enough depending upon your knowledge. I don't know much about Macs.
https://superuser.com/questions/581108/how-can-i-track-and-log-cpu-and-memory-usage-on-a-mac

[Lo Zeno]

Thanks! I’ll try this for sure. As a developer myself, it’s no problem.

You can also use top and htop like you would on linux (well... almost, top on mac is not at feature parity with top on linux), piping it to a text file. If I remember correctly, top is essentially what the Activity Monitory UI uses under the hood.

[BushViper]

It's great to see this getting some attention and (maybe?) some momentum.

My session is tonight and I'll have more concrete data to draw from.

[Makuzi]

Were having the same issue in our game. Except we have noticed that the lag is only present after someone moves a token. Doesnt matter if its via requested move then DM approves or if you use the arrow keys or if you are dragging it. The lag happens whenever a player or DM move their tokens.

[LordEntrails]

Were having the same issue in our game. Except we have noticed that the lag is only present after someone moves a token. Doesnt matter if its via requested move then DM approves or if you use the arrow keys or if you are dragging it. The lag happens whenever a player or DM move their tokens.
Please see post #4 (https://www.fantasygrounds.com/forums/showthread.php?71518-FGU-Memory-Leak&p=628956&viewfull=1#post628956) for the details to include that are needed to help investigate and resolve this issue.

[Makuzi]

Please see post #4 (https://www.fantasygrounds.com/forums/showthread.php?71518-FGU-Memory-Leak&p=628956&viewfull=1#post628956) for the details to include that are needed to help investigate and resolve this issue.


The way we fixed it for tonight is the DM turned off lighting and left line of sight on.

[BushViper]

I'm going to write a more extensive post later tonight when I get home from work, but I learned quite a bit during my session last night.

As predicted, memory was an issue, but something that I found incredibly surprising was my CPU activity. It got hammered hard when accessing data or interacting with almost anything (map, tokens, modules, etc). My CPU was pegged at 100% or close for most of the session and I had NOTHING but FG and Discord running. In fact, before the game, I stopped every single process that wasn't 100% necessary. I never even had an internet browser open. I kept the test bed as sterile as I possibly could.

Anyway, more later.

[Nylanfs]

Unity (and by extension FG) is single threaded I think, did you have it show all the cores?

[seansps]

Hi everyone. So during today's session, I did run FG using Syrupy to try to get some metrics. I had it poll the process every 1 minute.

I noticed that this session the lag did not get NEARLY as bad as it usually does. It was tolerable by the 4th hour and still mostly usable. I will note however there was a bit less rolling and use of the chat output this session than usual (but I am not sure how much that has to do with it.)

I've attached the output from Syrupy. You can see that it starts right off at the beginning with 4.9 GB of RAM usage... and it went up 9.9 GB by the end. You can see it continually go up and up. This time, it didn't use all my available RAM, but, it sure seemed to be going that way.

[Lo Zeno]

Hi everyone. So during today's session, I did run FG using Syrupy to try to get some metrics. I had it poll the process every 1 minute.

I noticed that this session the lag did not get NEARLY as bad as it usually does. It was tolerable by the 4th hour and still mostly usable. I will note however there was a bit less rolling and use of the chat output this session than usual (but I am not sure how much that has to do with it.)

I've attached the output from Syrupy. You can see that it starts right off at the beginning with 4.9 GB of RAM usage... and it went up 9.9 GB by the end. You can see it continually go up and up. This time, it didn't use all my available RAM, but, it sure seemed to be going that way.

Do you have the FGU logs of that session as well? I figure that it will be more useful to pair the syrupy logs to the app logs, to see which actions are linked to the increase of memory usage.
Also, even though I'm not part of SmiteWorks, I'm very curious to see what areas contribute to the memory allocation

[seansps]

Do you have the FGU logs of that session as well? I figure that it will be more useful to pair the syrupy logs to the app logs, to see which actions are linked to the increase of memory usage.
Also, even though I'm not part of SmiteWorks, I'm very curious to see what areas contribute to the memory allocation

Shoot, no, I meant to grab logs and forgot. I knew I was forgetting something lol. I will try to grab more logs/data next time.

[LordEntrails]

I wonder how large are people's campaign chatlog.html files? These files are written to by the chat and are not rotated etc, they live for the duration of the entire campaign. I now with other applications (SQL, Apache, etc) the logs are rotated or dated so that they do not get too large, because large log files can slow down the application.

Does someone want to rename their chatlog before their next session and see if that helps?

[seansps]

I wonder how large are people's campaign chatlog.html files? These files are written to by the chat and are not rotated etc, they live for the duration of the entire campaign. I now with other applications (SQL, Apache, etc) the logs are rotated or dated so that they do not get too large, because large log files can slow down the application.

Does someone want to rename their chatlog before their next session and see if that helps?

Interesting theory! I checked my original campaign, the chatlog.html file there is 2.9 MB. The lag in that one was also much worse. I created a whole new campaign for the final parts of the adventure module due to that one becoming too large (after creating modules for my custom content.).

My current chatlog.html file is just under 300 KB. I could definitely try renaming it before next session.

[seansps]

I'm not sure how useful it is, but I tried running the `leaks` command in macOS against FG. It seems like it believes there's a leak for all the tokens I've added to my `SmiteWorks/Fantasy Grounds/tokens/` folder as well as other files such as `db.xml`, and images in the campaign folder.

I also tried rolling a bunch of dice and some chat input before running the command. You can see at the bottom of the file, lots of random memory addresses. Could be related to the testing I did after initial load, but, I have no way to prove correlation short of having the source code and attaching a debugger/running something like valgrind.

(Split output into 2 files due to file size restrictions.)

[damned]

I wonder how large are people's campaign chatlog.html files? These files are written to by the chat and are not rotated etc, they live for the duration of the entire campaign. I now with other applications (SQL, Apache, etc) the logs are rotated or dated so that they do not get too large, because large log files can slow down the application.

Does someone want to rename their chatlog before their next session and see if that helps?

I dont believe this is loaded into the game at all.
I have some very large chat logs and they dont appear to affect the next session.

[Weissrolf]

Chatlogs can be edited and deleted mid-session, so they do not seem to be kept open/loaded, they are just written to.

[Weissrolf]

I've attached the output from Syrupy. You can see that it starts right off at the beginning with 4.9 GB of RAM usage... and it went up 9.9 GB by the end. You can see it continually go up and up. This time, it didn't use all my available RAM, but, it sure seemed to be going that way.
What makes your FGU use so much RAM? Is this specific to the Mac OS version of FGU or something about your campaign/extensions?

Your session start at close to 40 GB virtual memory on top of those 5 GB. My (Windows) sessions starts at 15 GB virtual memory on top of 1.6 GB physical, which I already consider too much to begin with.

[Zacchaeus]

Wouldn't the amount of memory at opening depend on the campaign? If you have a lot of stuff in the campaign, lots of open modules etc then I'd imagine memory would be a higher. The campaign that I have open as I write this is using 669Mb with no open modules. My PF2 campaign opens at around 1Gb because I have the adventure I'm running and the Core rulebook open and have some shared images.

[seansps]

What makes your FGU use so much RAM? Is this specific to the Mac OS version of FGU or something about your campaign/extensions?

Your session start at close to 40 GB virtual memory on top of those 5 GB. My (Windows) sessions starts at 15 GB virtual memory on top of 1.6 GB physical, which I already consider too much to begin with.

Interesting. Not sure. I have a bunch of modules loaded and images shared. But I don’t understand why that would cause so much RAM usage. Unless the image is open or a client is requesting it, why should it? Are all modules loaded into RAM too? Even so, it shouldn’t be causing a leak.

[Weissrolf]

Images inside modules are not loaded to memory until you open them for the first time, albeit loading (and unloading?) a module already eats some memory even with nothing but a single image inside. I created a test module with a single 19200 x 25200 px JPEG file, compressed JPG size 471 mb, uncompressed RGB size 1384 mb.

Empty PF2E campaign (no modules loaded): https://i.imgur.com/LtuW0Du.png

Image module loaded: https://i.imgur.com/zpHZVCe.png

Image opened: https://i.imgur.com/yh55OsW.png

Image closed: https://i.imgur.com/0TQeauL.png

One minute later: https://i.imgur.com/0rxOOaD.png

Five+ minutes later: https://i.imgur.com/UsgfTCC.png

Module unloaded: https://i.imgur.com/bG3jUEs.png

Return to Launcher: https://i.imgur.com/Axzccnn.png

No further comment, else this thread gets locked like so many before.

[seansps]

Images inside modules are not loaded to memory until you open them for the first time, even though loading a module already eat memory even with nothing in it but a single image. I created a test module with a single 19200 x 25200 px JPEG file, compressed JPG size 471 mb, uncompressed RGB size 1384 mb.

Empty PF2E campaign (no modules loaded): https://i.imgur.com/LtuW0Du.png

Image module loaded: https://i.imgur.com/zpHZVCe.png

Image opened: https://i.imgur.com/yh55OsW.png

Image closed: https://i.imgur.com/0TQeauL.png

One minute later: https://i.imgur.com/0rxOOaD.png

Five+ minutes later: https://i.imgur.com/UsgfTCC.png

Module unloaded: https://i.imgur.com/bG3jUEs.png

Return to Launcher: https://i.imgur.com/Axzccnn.png

No further comment, else this thread gets locked like so many before.

Very interesting. Is that… 8 Gigabytes of ram usage… after opening the image??

It would be a clue then as to why I used so much RAM last session but didn’t lag as much as before. I only alternated between 2 maps. I usually do 4 or 5 (if the party is traveling or have random encounters.)

The fact that the RAM is not free’d after the image is closed, to me, indicates that is the culprit. Hard to say, but I have a sneaking suspicion there’s at least one leak or issue with regards to images and how that memory is managed.

Edit: And if you look at my leaks files, the first one, it mentions a lot about what seems to be display related drivers. And of course, images.

[Lo Zeno]

Images inside modules are not loaded to memory until you open them for the first time, albeit loading (and unloading?) a module already eats some memory even with nothing in it but a single image. I created a test module with a single 19200 x 25200 px JPEG file, compressed JPG size 471 mb, uncompressed RGB size 1384 mb.

Empty PF2E campaign (no modules loaded): https://i.imgur.com/LtuW0Du.png

Image module loaded: https://i.imgur.com/zpHZVCe.png

Image opened: https://i.imgur.com/yh55OsW.png

Image closed: https://i.imgur.com/0TQeauL.png

One minute later: https://i.imgur.com/0rxOOaD.png

Five+ minutes later: https://i.imgur.com/UsgfTCC.png

Module unloaded: https://i.imgur.com/bG3jUEs.png

Return to Launcher: https://i.imgur.com/Axzccnn.png

No further comment, else this thread gets locked like so many before.

Can you attach the module? I wanna run the same steps myself on my PC and my Mac.

[Weissrolf]

Very interesting. Is that… 8 Gigabytes of ram usage… after opening the image??
Indeed.


The fact that the RAM is not free’d after the image is closed, to me, indicates that is the culprit. Hard to say, but I have a sneaking suspicion there’s at least one leak or issue with regards to images and how that memory is managed.
It makes sense to not remove every image from memory the moment its window is closed, else they would have to be reloaded every time (takes a lot of time to load this large image). I mean to remember that once I saw memory being unallocated after a few minutes, but in this test 5+ minutes obviously were not enough.

Unloading the whole module also does not release the image from memory. I can re-load the module after some minutes again and open the image and it is still displayed immediately from memory, instead of loaded from disk.

There also is the problem of (every 5 minute) Autosave messing with FGU's memory management (and maybe other things) when saving happens while FGU is busy doing other stuff. So this may play into this on top of already high usage.

Memory not being freed up when FGU is returned to its Launcher is another red flag. Everything about this is known for quite some time, but these reports are not understood as pointing to a general issue, but always only connected to single (test) usage cases. So don't get your hopes up too much.

[Weissrolf]

Can you attach the module? I wanna run the same steps myself on my PC and my Mac.
https://1drv.ms/u/s!AsEwNk439-yxi6FdnsB1K5Drm0P0oQ?e=gsPxRb

Link active until 18-12-2021.

[Weissrolf]

I waited some more minutes after unloading the module and did other stuff at the computer. FGU's memory load decreased from 8 GB to 6 GB. Re-loading the module and image then had the image being loaded from disk again with FGU's memory increasing back to 8 GB. So the image was removed from memory at some point, but FGU kept the memory allocated. Once the image was re-loaded FGU did not allocate new memory, but reuse the already allocated memory.

This is similar to what I remember to have reported quite some time ago in some other thread. The wildcard here likely is Autosave, because from other tests I know that it can mess with FGU's memory management.

[Weissrolf]

Loading the same image in...

Irfanview: https://i.imgur.com/E7yI1ye.png

Photoshop: https://i.imgur.com/K0Tz6iK.png

Anyway, that was more than I intended to post. You will have to handle this on a single case basis with Smiteworks, they don't believe in bigger pictures solutions when it comes to FGU performance.

[Lo Zeno]

https://1drv.ms/u/s!AsEwNk439-yxi6FdnsB1K5Drm0P0oQ?e=gsPxRb

Link active until 18-12-2021.

Downloaded, thx

[Zarestia]

@seansps It might be better to share your campaign or e-mail to SW if you've got 4.9 GB RAM usage at the start of the session, that seems unnatural.

More of a real life example instead of some theoretical bingo:

Dev Campaign opened, DB size: 4.08 MB, 75 images, 43 MB all images in sum, all jpeg
~600 MB RAM usage at the start
~1350 MB RAM usage after opening and closing all 75 images
~ 850 MB RAM usage after ~5min

In over 2 years of playing I haven't gotten over 2 GB RAM usage even after 5-6 hours. A look into the specific campaigns is necessary... otherwise we're running circles.

[seansps]

@seansps It might be better to share your campaign or e-mail to SW if you've got 4.9 GB RAM usage at the start of the session, that seems unnatural.

More of a real life example instead of some theoretical bingo:

Dev Campaign opened, DB size: 4.08 MB, 75 images, 43 MB all images in sum, all jpeg
~600 MB RAM usage at the start
~1350 MB RAM usage after opening and closing all 75 images
~ 850 MB RAM usage after ~5min

In over 2 years of playing I haven't gotten over 2 GB RAM usage even after 5-6 hours. A look into the specific campaigns is necessary... otherwise we're running circles.

If the devs ask for that, I can provide it. I also have a custom module I made with additional maps, which is loaded too. (The .mod file is 61 MB). I assume that would probably be necessary, otherwise, it isn't really the same test scenario. However, I didn't load or share any map from that module last session. The campaign I am playing is Princes of the Apocalypse, in case that matters.

[seansps]

I'd also like to add that I don't think I am doing anything typically "out of the norm" for a use case of FG. I make characters, add portraits and tokens, maybe make a few Story entries of my own, share maps, run encounters, roll dice, load modules for 5e. The only weird thing I do is perhaps include a ton of tokens directly in the Fantasy Grounds/tokens directory in case I want to improvise an NPC/creature/etc.

[Lo Zeno]

@seansps It might be better to share your campaign or e-mail to SW if you've got 4.9 GB RAM usage at the start of the session, that seems unnatural.

More of a real life example instead of some theoretical bingo:

Dev Campaign opened, DB size: 4.08 MB, 75 images, 43 MB all images in sum, all jpeg
~600 MB RAM usage at the start
~1350 MB RAM usage after opening and closing all 75 images
~ 850 MB RAM usage after ~5min

In over 2 years of playing I haven't gotten over 2 GB RAM usage even after 5-6 hours. A look into the specific campaigns is necessary... otherwise we're running circles.

On what OS?
FGU has a VERY different RAM usage in Windows vs MacOS: when I open my Dev campaign in Windows, at the start it's using 547 MB of RAM; if I open the same campaign on MacOS, it's a whopping 2453 MB of RAM.
This is, I suspect, at least in part due to how Unity implements certain common patterns differently for each OS.

[seansps]

On what OS?
FGU has a VERY different RAM usage in Windows vs MacOS: when I open my Dev campaign in Windows, at the start it's using 547 MB of RAM; if I open the same campaign on MacOS, it's a whopping 2453 MB of RAM.
This is, I suspect, at least in part due to how Unity implements certain common patterns differently for each OS.

Good point Lo Zeno! I am on macOS, and seeing these crazy RAM usages. If the answer for me to get better performance for now is to run it in Windows... I can switch to my windows machine. Though I'd prefer not to have to!

[Weissrolf]

Might use less memory via Parallels than using the native FGU macOS app then.

[Zarestia]

The only weird thing I do is perhaps include a ton of tokens directly in the Fantasy Grounds/tokens directory in case I want to improvise an NPC/creature/etc.

There have been problems with the token directory in the past. I'd have to search the thread, but it was about a few weeks ago.


On what OS?
Windows 10, this was in response to Weissrolf. As it seems that FGU does a way better job at handling and clearing memory with many "normal" images instead of one nearly 500 MB image.


If the answer for me to get better performance for now is to run it in Windows... I can switch to my windows machine. Though I'd prefer not to have to!

RAM usage doesn't correlate per se to performance. You could of course test it in Windows and report back whether you see the same performance issues.

[BushViper]

I've neglected responding with my own session results in my own thread because, the day after my game, I was transferring the files from my PC to my phone and and accidently deleted them.

I haven't come back here out of sheer embarrassment, but avoiding it isn't going to solve anything. I have another session coming up and I'll be more 'responsible.'

That said, my own experiences and reading about numerous others, it seems that FGU is just wildly inefficient in managing both RAM and CPU load.

[Weissrolf]

I use successfully use Onedrive since the beginning days of using FG, which also means that "accidentally deleting" files while transferring them anywhere is not a problem. Of course I also have backups.

Prepare for incoming...3...2...1...

[Weissrolf]

RAM usage doesn't correlate per se to performance.
Well, if your software uses six (6!) times the memory necessary for a raw data object then I bet its single-threaded poor CPU process also somehow has to deal with all that extra data. :bandit:

[seansps]

I've neglected responding with my own session results in my own thread because, the day after my game, I was transferring the files from my PC to my phone and and accidently deleted them.

I haven't come back here out of sheer embarrassment, but avoiding it isn't going to solve anything. I have another session coming up and I'll be more 'responsible.'

That said, my own experiences and reading about numerous others, it seems that FGU is just wildly inefficient in managing both RAM and CPU load.

I was wondering when we’d get your update! Haha, it’s okay, I’ve definitely done something like this before.

Looking forward to see if you’re able to find anything as well, but I agree, there’s something off with how RAM is managed. I’ve never used FGC, having come straight into FGU… but when I see videos on YouTube of FGC it just seems so much more smooth and less sluggish.

Anyway, I love FG and really don’t want to have to switch to Foundry (which I’ve tried and the automation just is not there and is a mess of modules/3rd party addons, and no official 5e content.) So, I’m really hoping between both of our findings we can help better the FG platform. I’ll continue to try to gather data, too.

Edit: And yes I know about Roll20, and Astral, and Owlbear.rodeo and everything else, and they all suck. I legitimately want to help FG be the best, as it already is, imo.

[BushViper]

I was wondering when we’d get your update! Haha, it’s okay, I’ve definitely done something like this before.

Looking forward to see if you’re able to find anything as well, but I agree, there’s something off with how RAM is managed. I’ve never used FGC, having come straight into FGU… but when I see videos on YouTube of FGC it just seems so much more smooth and less sluggish.

Anyway, I love FG and really don’t want to have to switch to Foundry (which I’ve tried and the automation just is not there and is a mess of modules/3rd party addons, and no official 5e content.) So, I’m really hoping between both of our findings we can help better the FG platform. I’ll continue to try to gather data, too.

Edit: And yes I know about Roll20, and Astral, and Owlbear.rodeo and everything else, and they all suck. I legitimately want to help FG be the best, as it already is, imo.

I have no experience with FGC either, but everyone says it runs far better. However, I have no desire to use it because it's trash (not to mention no longer supported) compared to FGU even with the performance issues. I, too, want FGU to be the best it can be because the platform is great and has SO MUCH potential to be the undisputed king of VTTs, but it's not there yet.

I completely agree about the other VTTs; Roll20 is hot garbage. I'd rather not play D&D online at all than use Roll20. Astral and Owlbear serve a purpose to a niche crowd, but they're essentially just a battlemap. That just doesn't do it for me.

Foundry is fantastic and has A LOT going for it, but it also has some major drawbacks. For one, it requires a TON of maintenance for it keep functioning as you expect it to. The ext/mod creation community is basically the Wild West. Stuff has constant conflicts, gets abandoned, and some of the content occupies a legally gray area that I'd rather not be in. Still, if FGU doesn't get its stuff together they're going to eventually lose a bunch of its users (me included) when Foundry matures or another well-backed VTT comes on the scene. Believing that another big-baller VTT is unlikely to happen, is simply naïve.

[seansps]

I gathered some more data after my session yesterday, which lasted just over 4 hours.

I noticed again that, around the 4 hour mark, I started to see lag with visuals (rolling of dice, the map, etc.) However, like before, it was not as bad as it has been. It was still noteworthy though, and I don’t think the session could have gone much longer without it getting unbearable.

I took a screenshot of the memory state of the application from Activity Monitor immediately after loading the campaign, then, 40 min later, after I prepared everything for the session, and then again at the end of the 4.5 hour session. You can see that the memory is going up quite a bit (and a lot of virtual memory usage) but not as bad as it’s been in the past.

I also compiled the logs, this time. Hopefully it is helpful. I haven’t looked much at them, but I do see a lot of Null Reference Exceptions I think to do with images, not sure.

[Moon Wizard]

I have another build queued up that has some small improvements to the UI control queueing behaviors and cleanup, based on some memory analysis I did internally. It won't be released for about 1-3 weeks. I'm sure there's more to do; but it should incrementally help once it's out.

Regards,
JPG

[seansps]

I have another build queued up that has some small improvements to the UI control queueing behaviors and cleanup, based on some memory analysis I did internally. It won't be released for about 1-3 weeks. I'm sure there's more to do; but it should incrementally help once it's out.

Regards,
JPG

That’s great news! I look forward to testing when released. Thanks for the update!

[seansps]

Hi all, I'm back with another update.

Unfortunately I don't have any data to show for this time, as nothing seemed to stand out. Basically, last night, I had 4 players connected and it was another 4 hour session. Everything was smooth and great until near the 4th hour, I'd say about 3.5 hours in, the session started to get really, really choppy. First thing I did was try to look at the memory usage, but it looked about the same as my previous reports of 4-hours use of memory (4+ gigs used.)

I noticed several things, however.
1. Immediately I realized some of the lag was due to save lag (the same bug I reported before, that is caused with drawing on the map.). I reverted changes to the map and that cleared that up. But,
2. It didn't clean up the choppy-ness of the performance.

The lag is generally slow/unresponsive UI:
- Moving tokens (by clicking, or shift-clicking and dragging) on the map became extremely slow (it was very very hard to move them anywhere.)
- All dice animations come to a choppy halt (sometimes not even showing up) and then a long delay between the roll and the result. (Player, "Did it hit?" Me: "Sorry... waiting for the lag! It's a.... miss... no hit!")
- Players reporting difficulty moving on their end as well as graphical issues with the map.
- Dragging windows is slow and choppy
- Clicking-and-dragging on a link, it takes a while for the link to show that it is being dragged, so that takes longer too

I am confident that closing and reopening FG will resolve all the lag but I really don't like doing that to players (especially in the middle of a combat) but I will try this next time to confirm.

As before, I was running:
- Fantasy Grounds
- Chrome
- Discord

Curiously, I noticed that as players started disconnecting, my UI seemed to be more responsive again, but I didn't try too much after the end of session.

[Moon Wizard]

I did some analysis last week and found that there was a fair amount of overhead in FoW data. I have the save speed bottleneck fix for FoW in the next release (soon).

Also, @cpinder is currently out of office; but I plan to work with him to see if we can streamline the FoW data, which can get quite large. Small LoS sections such as per bar portcullis or lots of small pillars can exacerbate the situation.

The save impact will occur even for closed maps, since the data is still stored and reloaded; so any edited map with tokens would need to be reverted or tokens removed. The save fix I have in progress will help mitigate the save time but the data can still get large.

Note that the performance impact will remain until FoW is cleared for each token, and closed/reopened. (Or reverted and closed/reopened). This is because the FoW data is preserved in undo states while the image is open as well. (And cleared on close)

More than likely, any changes we make to optimize the FoW will probably reduce the fidelity of the edges slightly to simplify the shape and number of points.

On a related note, extensions that attempt to preserve fog of war between maps may make the situation worse, since the large data sets will be getting duplicated. I recommend turning these off for now.

Regards,
JPG

[seansps]

Thanks for the update Moon - that sounds very promising! Looking forward to those optimizations.

[Sterno]

Any chance while you guys are in that code you'll add an option to just disable FOW entirely? *hopeful smile*

[Jiminimonka]

Any chance while you guys are in that code you'll add an option to just disable FOW entirely? *hopeful smile*

Not a bad idea for the Options section

[Moon Wizard]

It's something that's on our near term list; but it requires a fair amount of work in the client to make sure it works correctly and that the network/file serialization is in place. Also, it requires APIs, so that it can be turned on/off in the rulesets appropriately; as well as ruleset code updates to use the APIs. I'll have to wait until @cpinder returns to work with him on it.

Regards,
JPG

[jharp]

I did some analysis last week and found that there was a fair amount of overhead in FoW data. I have the save speed bottleneck fix for FoW in the next release (soon).

Also, @cpinder is currently out of office; but I plan to work with him to see if we can streamline the FoW data, which can get quite large. Small LoS sections such as per bar portcullis or lots of small pillars can exacerbate the situation.

The save impact will occur even for closed maps, since the data is still stored and reloaded; so any edited map with tokens would need to be reverted or tokens removed. The save fix I have in progress will help mitigate the save time but the data can still get large.

Note that the performance impact will remain until FoW is cleared for each token, and closed/reopened. (Or reverted and closed/reopened). This is because the FoW data is preserved in undo states while the image is open as well. (And cleared on close)

More than likely, any changes we make to optimize the FoW will probably reduce the fidelity of the edges slightly to simplify the shape and number of points.

On a related note, extensions that attempt to preserve fog of war between maps may make the situation worse, since the large data sets will be getting duplicated. I recommend turning these off for now.

Regards,
JPG

I will also say that I found the following in my FoWEnhanced testing. Maybe the FoW lines should be shorter lines with more lines. Again, Moon, I'm open to suggestions on improvements to FoWEnhanced.


Interesting Tidbits

If you save a single DB entry of 84k characters the save is several 5+ seconds but with 84 1k entries its less than 1/2 second to save.
A 15k character entry saves 10 times faster per character than does a 85k character entry.

[jharp]

On a related note, extensions that attempt to preserve fog of war between maps may make the situation worse, since the large data sets will be getting duplicated. I recommend turning these off for now.

Regards,
JPG

On this single point of Moon's, FoWEnhanced does not store the FoW data in the DB so the data is not duplicated in the DB. Obviously strings are used in the process of getting the FoW data, and these will remain around until collected. I see no reason for turning off FoWEnhanced at this time. If Moon has additional data to share, I'm happy to evaluate if FoWEnhanced is a significant contributor to the issue.

Jason

[Moon Wizard]

If someone is running their FG folder off of a network or external drive, the file load/save times could add potentially significant delays to any archiving and restoring process. So, while your extension may not save the data in the FG database, it is storing the data in a file somewhere on the disk.

Additionally, if there are large FoW data sets that have built up, and your extension is restoring those large data sets; then it propagates the large data set issue. While it is something that I want to look at, it is not something that we will be able to immediately address.

Between those two items, that is why I suggest not running with the extension. It's not that the concept is bad or unwanted; but the system will not handle it well, especially in certain situations.

Regards,
JPG

[jharp]

Well I guess the choice is to not have the feature for quite some time or to have the feature now with some slow down (if any). Please if anyone is having slowdown with the save and load of FoW data let me know directly on the FoWEnhanced support thread.

We can only dream of a time where FG actually works better out of the box and doesn't need a bunch of extensions to fill out the functionality. Personally, I don't think anyone should have to pay for the functionality provided by FoWEnhanced. It should have been built into FG to start with.

Jason

[BushViper]

Disclaimer - I'm not a programmer so the language I use may not be correct, but the hope is that the idea is generally understood.

@Moon Wizard.

So, I've been doing a lot practical testing to see what I can learn (and possibly do) about the performance issues, myself. While I haven't discovered anything new, my testing has raised some questions, for me.

Line of sight, lighting, shadows, etc. all have massive performance costs. As do extensions that need to consider the positions of tokens, occluders, etc., on a map.

So, my question is this -- Why does the FGU engine constantly calculate the data for everything, all the time? Instead of....say....just the token(s) that are selected/targeted?

Admittedly, I don't know exactly how the data is handled, but I have been tinkering with the platform for long enough that, at least to me, it seems that there is a bunch of redundant, or unnecessary processes/calculations happening, which absolutely crush performance.

I don't know. I could be completely off the mark, but every time I log in to FGU I'm left scratching my head because I literally cannot begin to understand how, for what amounts to, a relatively rudimentary GUI has its performance brought to its knees by a few tokens, some extremely basic lighting, and shadows.

Like everyone else, I get plenty of the pre-described "lag" where mod files take 3-10 seconds to open and, every session, I get my share of whiteouts that end up as "not responding" hang-ups before FGU raises itself from its temporary death. While I hate that crap, I can live with it. However, the absolute garbage performance on maps is what's going to completely poison the well for me.

Performance issues are becoming an increasingly common topic of discussion in the numerous FGU Discord servers I'm a member of and, as far as I can tell, a lot of people are getting completely fed up with having to make feature concessions, spend days Googling for work-arounds, and then having to devote even more time to figuring out how to implement those 'fixes.' All just to get a half-baked session out of the platform. Unless, of course, the user is running a client with a tiny campaign folder, not using LoS, no lighting, no shadows, no extensions, keeping map/token size small, and only load the mod files that are absolutely necessary for your campaign. Do all that and your client will hum right along. Sounds amazing...to uninstall.

FGU has a steep learning curve, but it's rewarding once you get ahead of that curve. However, many people will never get there because they're not going to invest the time and effort it takes to master FGU when the UI looks and runs like it's on a Nokia cell phone from 2002.

I've said it before and I'll say it again. I truly like FGU and I want it to succeed, but people have limits to what they're willing to tolerate and I don't think that expecting the platform to run without constant, debilitating interface lag is too much to ask. I also understand that SW is a small company and you only have so many resources to allocate to any particular problem, but from where I'm standing, it seems like addressing fundamental issues like performance trumps nearly everything else because no-one cares how many bells and whistles it has if it doesn't run well. However, according to many of your staff, "performance issues only impact a small minority of FGUs users," "are greatly exaggerated," or due to some other excuse.

I've had a number of sessions after accidently deleting the data I was going to turn over but, after a lot of thought, I decided I wasn't going to bother packing up my campaign folder so some forum mod could examine it and then instruct me how I could shuffle a few files around to squeeze out a negligible increase in performance. That does absolutely nothing to fix the fundamental problems and for those of us that can grasp what's really happening and have spent literal weeks trying to fix it ourselves, the hubris is infuriating.

[Moon Wizard]

Any time that a token, occluder, point light, or token light is moved; all calculations (what to draw on screen, intra-token visibility, LoS, FoW, etc.) have to be redone. Under the hood, all of that information is just points, lines and radii. (i.e. there is no implicit "regioning" of data like your eyes and brain do automatically) Additionally, the way that the graphics work for LoS/FoW/lighting/FX/etc is that they use GPU shaders to determine the value of a pixel on every frame. So, lots of data slows down per frame performance.

The particular issue that I think we're running into right now is that the fidelity of the fog-of-war is too high. It is mirroring LoS exactly down to the pixel; which creates very large data sets. That's why I mentioned we're looking at ways to simplify the data (similar to how other games use more blocky fog-of-war).

In no way would I describe the image systems implemented in FGU as "rudimentary GUI". If you are a developer, you are welcome to explore building out image mapping/exploration GUI systems yourself to understand the considerations. As for the other UI elements, this is a subjective matter, and personally I prefer theme-ability and customizability over cookie cutter UI.

As for performance in general, I am constantly mentioning to people to keep things simple, to keep maps under certain sizes, etc. These are guidelines to improve performance.

For specific performance on specific machines, you can always try setting the "/vsync 0" to force 60 fps, because I believe the ever increasing monitor frequency rates and the default tying of frame rate to monitor refresh rates to be slowly increasing the per frame rate calculation burden. After doing some recent monitor shopping, I'm considering whether we should force the fixed 60 fps frame rate by default; with perhaps an additional setting for fixed 30 fps for older machines.

As for providing the data and steps you followed, this is a very important part of the feedback process to improve the product. We are not a gigantic company with 100s of people developing this software; but just a small company with only 2 client developers and 2 ruleset developers (out of 12 people total). We are looking to build up, but that takes time to find the exact right people.

By providing data sets and reproduction steps, this allows us to more easily pinpoint specific scenarios where the current systems are not performing well, and may even allow us to pinpoint the exact code we can adjust, fix, or analyze. Without data/steps, we are essentially guessing which one of a gigantic number of moving parts "might" be the issue. Then, we spend all our time on investigation, instead of fixing issues.

If you are having issues with particular scenarios, you need to raise those issues. That being said, sometimes the answer will be to pare down the data (whether it's number of custom modules, extensions, etc.)

Regards,
JPG

[bmos]

The particular issue that I think we're running into right now is that the fidelity of the fog-of-war is too high. It is mirroring LoS exactly down to the pixel; which creates very large data sets. That's why I mentioned we're looking at ways to simplify the data (similar to how other games use more blocky fog-of-war).I'd be happy with fog of war that was 1/4 of the grid size, personally. Pathfinder rules seem like they'd be fine with full grid squares even, but 4-8x that resolution would be much prettier.
Right now it's possible for LOS to be calculated at over 100x the grid size on a high-res map which is (as you point out) quite excessive.

[Moon Wizard]

The problem with doing chunky blocks of fog-of-war is that it could inadvertently expose "more" than expected, and isn't necessarily what most people expect to see. I was thinking more about smoothing the curves, so that that small angles are combined into straight lines that reveal just "slightly" less, but significantly reduce data points. As I mentioned, this is something I need to explore with @cpinder, who is currently out of office, as well as on another project.

Regards,
JPG

[BushViper]

As for performance in general, I am constantly mentioning to people to keep things simple, to keep maps under certain sizes, etc. These are guidelines to improve performance.

Regards,
JPG

Heh. Christ, it's a lost cause, but I suppose I should thank you for reiterating one of the fundamental points of my post.

[LordEntrails]

Moon, just in case you and Carl have not thought about it already, but what about periodic refactoring the FoW to consolidate the data into polygons? You could perhaps set/control the resolution of the polygons to some factor of grid size. And maybe do it based upon action (open/close image?) or periodicallly.

No need to answer either way, just a thought that might have been worth your time.

[Kelrugem]

Heh. Christ, it's a lost cause, but I suppose I should thank you for reiterating one of the fundamental points of my post.

Errr, Moon Wizard actually wrote several times in his posts that they look at that, here FoW was mentioned; but you seemingly chose to ignore all of that. The advice to trim things down is first of all an advice until the performance is improved, to have something playable; and second, there will be always some soft limit of performance somewhere, also after performance improvements. Keeping things smaller than planned of course always helps, one could rephrase it as "do not overdo it" to avoid hitting limits, or "know your computer system(s) and its limits"

Saying "to keep things small" does not exclude "performance improvements", both things will always be important for performance; the former is something the user can do, the latter the devs which then helps to lift the former. But these are not mutually exclusive, so, I do not understand why you see that one statement as proof for that performance won't get improved?

[MrDDT]

Errr, Moon Wizard actually wrote several times in his posts that they look at that, here FoW was mentioned; but you seemingly chose to ignore all of that. The advice to trim things down is first of all an advice until the performance is improved, to have something playable; and second, there will be always some soft limit of performance somewhere, also after performance improvements. Keeping things smaller than planned of course always helps, one could rephrase it as "do not overdo it" to avoid hitting limits, or "know your computer system(s) and its limits"

Saying "to keep things small" does not exclude "performance improvements", both things will always be important for performance; the former is something the user can do, the latter the devs which then helps to lift the former. But these are not mutually exclusive, so, I do not understand why you see that one statement as proof for that performance won't get improved?

Fully agree with this post, SW clearly knows there is an issue and are working to solve it. I'm one that has a ton of EXTs running and I can choose to run without them or have it slowed down a bit and run with them.
The exts also are improving, not only are they improving but SW is working to give EXT writing more tools to work better and do things in a better manner often.
Rome wasn't built in a day and things are getting better. If you play FGU without any exts, it is really smooth, I have to say its not bad (just don't load up 200 spells on a PC sheet and expect to use the action tab =P) its really nice.
I've gotten to the point myself I can't live without many of the EXTs I use. Some do cause the game to slow down a bit, however, the option is there to have it.
On it's base FGU is still really fast and smooth and highly useful to jump in (after learning how to use FGU) and play D&D with a great deal of automation added to vanilla FGU. I think the key is not to despair and have hope that its coming along and give it time, report issues, give ideas and keep on going.
FGU is still so much better than FGC, lets not even talk about compared to other VTTs how much better FGU is.

[seansps]

FGU is still so much better than FGC, lets not even talk about compared to other VTTs how much better FGU is.

So much this. I keep checking in with Foundry and their progress. You still need like 10 modules to even come close to the level of automation FGU provides out of the box. Then to get all your 5e content in there you have to either manually handjam it or use a module to import content from another source like DnDBeyond. And FWIW I noted that the performance of Foundry on my MacBook Pro was sluggish beyond use on my Ultrawide monitor, which FGU handles just fine. So- I’m sticking to FG.

In any case I came back to report on my session this night. I noticed that the same lag came back after 3 hours of play. It was quite constant, and not to do with the autosave. I accidentally quit FG in the middle of the session while trying to get open Activity Monitor (lol.). But I re-opened the app and we continued for another hour.

I will note that as I expected, after closing and re-opening the application, all the lag vanished. So, my question is: Is this still indicative of FoW issues? Does FoW reset on close of the application?

[Trenloe]

In any case I came back to report on my session this night. I noticed that the same lag came back after 3 hours of play. It was quite constant, and not to do with the autosave. I accidentally quit FG in the middle of the session while trying to get open Activity Monitor (lol.). But I re-opened the app and we continued for another hour.

I will note that as I expected, after closing and re-opening the application, all the lag vanished. So, my question is: Is this still indicative of FoW issues? Does FoW reset on close of the application?
Are you using the live channel or the test channel?

There is now a public beta test that includes some changes for performance improvements, information here: https://www.fantasygrounds.com/forums/showthread.php?72524-FG-Unity-Beta-Release-v4-1-14 It might be worth you trying this for you next session?

[seansps]

Are you using the live channel or the test channel?

There is now a public beta test that includes some changes for performance improvements, information here: https://www.fantasygrounds.com/forums/showthread.php?72524-FG-Unity-Beta-Release-v4-1-14 It might be worth you trying this for you next session?

Interesting! I could try that next time, I use the live one to ensure I’m on stable builds.