DM games slows to a crawl

[webdove]

Last friday's game (FGU latest, DotMM, Lighting etc) started without issue at 7pm. Over the evening the DM game became slower and slower. Towards midnight the lag for the DM was 2-4 seconds between pressing an NPC attack or damage and being able to drag it. Double clicking vs targeted PCs no longer worked. Players did not report lag on their end. The Host PC was not resource limited.
https://i.imgur.com/77ghtW1.png

At the end of the session while were we talking in zoom I asked the players to disconnect. The host game immediately sped back up to normal response times. Then I asked them all (6 players) to reconnect and the host game still had no lag.

It behaved as though there was a communication delay that built up over the course of hours and cleared once the players disconnected.
Unsharing the map did not help.

[Zarestia]

Would be interesting what exactly hogged all that CPU & RAM usage.

- How big is your campaign's db.xml?
- Are many images shared (filter images as "shared")?

[Moon Wizard]

Also, which game system are you playing; and which extensions are loaded?

Some thoughts:

* New image behaviors slowing things down (which Zarestia's questions may help identify)

* Also, I've found that sometimes there is script code which actually creates network loops that are going on continuously; and can compound over time as more loops are created. I monitor my own ruleset code pretty closely to avoid this sort of issue; but not everyone building scripts understands to watch for those sorts of things. Once identified, I can fix (if it's my code) or help the developer (if it's a DLC developer or community developer).

Regards,
JPG

[webdove]

The db.xml is 6 megabytes. There are 29 old public images from earlier levels mostly pictures of creatures.

If that was the problem, wouldn't it manifest at the start of the game rather than building up over the course of the evening?

https://i.imgur.com/Q7Rh5GY.png

[webdove]

The db.xml is 6 megabytes. There are 29 old public images from earlier levels mostly pictures of creatures.

If that was the problem, wouldn't it manifest at the start of the game rather than building up over the course of the evening?

I am using extensions derived from Mattekure's google drive extensions list. (see below).

I am using the FG-PFRPG-combat-timer.ext which pings and pongs as a way of keeping time. https://github.com/Morturg/FG-PFRPG-Combat-Timer/releases
I added "5E" to the allowed rulesets so I could use it in my FGU/DotMM campaign.
I timed it once using os.time() and it seems to ping/pong at about 10 hz.


Is there a way to inquire how many network loops exist as the game progresses?

https://i.imgur.com/2wbz9BU.png

[Moon Wizard]

A db.xml with 6MB of data is pretty large for most campaigns; since all of the mask data for any shared/modified module data is stored in the moduledb folder instead.

Network loops can't be detected easily, because it's completely determined by the logic of the code causing the looping behavior. If client A changes a value to 1, which triggers a network update; and client B senses the change, and has code to change the value to 0; which client A senses and changes back to 1; then there is a network loop. There is no automatic mechanism to identify that sort of behavior.

Just the fact that you have over 50 extensions in your folder (and probably loaded in your campaign as well) leads me to believe that this is most likely an extension issue. Try disabling your extensions, and seeing if the default behavior in the plain ruleset is better.

Regards,
JPG

[myyra]

I noticed something while making maps today, and it might be related to this problem. My FGU seems to be leaking a lot of memory when I have lighting turned off in a map I'm making. It's a huge map, and I probably shouldn't be making anything so big, but these things happen...

Anywho, when I have Lighting enabled, the memory used by FantasyGrounds.exe remains stable, however, the moment I disable lighting the memory consumption starts going up by about 2-3MB/s and that will add up quickly. Has increased by about 800MB in less than 10 minutes. Enabling the lighting halts the progress immediately.

I'm using the PF2E ruleset and working on a stupidly large map. I will take a look if I can reproduce this with some other settings later, but cba to do it right now, because it's super late and I should be sleeping.

[Moon Wizard]

Can you provide a zipped up copy of campaign with that map where you can give us steps to exhibit that issue?
(You can provide link to Google Drive or DropBox; since I'm going to assume it will be a large file.)

Thanks,
JPG

[webdove]

Here is the top level table of contents of my DotMM db.xml with line numbers. Does anything wonky jump out?
https://i.imgur.com/KNKW8JB.png

[webdove]

There are large numbers of empty "<ShiftedState>" entries
https://i.imgur.com/Rg7HNo5.png

[webdove]

Charsheets are most of it. There are 15 characters. A few are backups for the wizards so they can play a version that only holds prepared spells (for better speed). 3 are utility characters for my experiments.
https://i.imgur.com/LZn6lRq.png

[webdove]

We only have 8 characters in the game at one time. One _PartyVision and 7 players.

[Moon Wizard]

@webdove,

I'm not going to be able to guess about behaviors of 50+ extensions that I did not write and I am not familiar with as to which ones might be slowing down performance. Just looking at the db.xml example; I can tell at least half the data in that file is related to some extension or another (since they are using tags not used by the base ruleset code).

As I mentioned before, I highly suggest trying to run sessions with NO extensions and see if that's still an issue. I know that this is probably not what you want to hear; but we do not have resources to review and debug every community extension ever made, especially with all the permutations of running them all at once.

Regards,
JPG

[webdove]

I completely understand and I appreciate your advice. One final question that is I think more appropriate. I need some advice on where to look for the software objects that contain currently applicable attack "reach" information for PCs and NPCs. Where should such a question be posted?

[Trenloe]

I need some advice on where to look for the software objects that contain currently applicable attack "reach" information for PCs and NPCs. Where should such a question be posted?
Best to post system specific information in the sub-forum for the RPG system you're using. Forum -> Game Systems -> <Your RPG System>

[webdove]

Thanks Trenloe

[webdove]

Also, which game system are you playing; and which extensions are loaded?

Some thoughts:

* New image behaviors slowing things down (which Zarestia's questions may help identify)

* Also, I've found that sometimes there is script code which actually creates network loops that are going on continuously; and can compound over time as more loops are created. I monitor my own ruleset code pretty closely to avoid this sort of issue; but not everyone building scripts understands to watch for those sorts of things. Once identified, I can fix (if it's my code) or help the developer (if it's a DLC developer or community developer).

Regards,
JPG

I don't think it is image memory or memory in general since I still have memory reserve. In fact I seem to have reserve in cpu and gpu as well.

Given that, I like your theory about network linkages to client objects accumulating over time and that causing increased delays.

Am I correct in assuming that all helper callbacks originate at the host and communicate to a client while waiting for a response? If that is true then if the clients are for some reason adding helper callbacks over time that could increase host network handshaking delays.

Perhaps I could observe the network sends and receives as a function of time. Wireshark would help me isolate which client if it was class dependent.

[Temmpest]

Isn't there a command the players can enter into the chat box to help with lag?

[Moon Wizard]

No, there is no way to know whether the network traffic being sent is valid or not. FGU is a client platform for ruleset and extension code that just passes along state information and database information to each client as updates occur.

As far as I am aware, there are no network loops in the base code for the rulesets included in the standard install; but I'm willing to work with anyone who has evidence otherwise. However, we do not take any responsibility for third party add-ons.

@webdove,
Callbacks can happen on either the GM or player machine depending on how the scripts are written. Player clients can only see data which is owned by the player or shared with the player; and can only modify data which is owned. Plus, there are some APIs which are host only. Other than that, the behaviors should be similar for scripts running on each end.

Regards,
JPG

[webdove]

I can use wireshark "udp && udp.port ==1803" to watch my packets then use I/O graphs.

With a single PC, 2 NPCs, one client and no extensions it seems to run about 10 packets/second increasing to 70 packets/second if I drag the PC around in circles on the map.

I will try this out on our game this Friday.