Latest test game in TEST server - that was painful

[SilentRuin]

That was painful. I just ran a 3 hour attempt to run with TEST server on my FGCON16 map tool art subscription assets created maps with lighting.

Last week I ran through the same maps and had a few blackout issues. This week - I had a ton of blackout issues and then horrific eventually unplayable random lag followed by total lag for everyone.

It was a LAN game. I would see the things happen instantly that a player did - but they would not see the player rolls until much later. As time progressed it became unplayable for us all. Some were like 2 minutes behind on things that had happened. Even after I shutdown the maps.

Here is what we found:

When a light equipped player did an end turn, random blackouts. One or multiple players and it varied. Sometimes it would be fixable by resharing the map. Sometimes it would be fixable by player closing map and reopening it from images. Sometimes they had to logout and back in. Sometimes only removing the token and CT entry and replacing it entirely would make it work. Sometimes the DM had to exit the game entirely when none of those things worked. Never happened when a non light effect CT entry had end turn.

A player with a torch had waypoint movement lag issues with a torch.

A player with a torch only saw grey (darkvision) even with a torch effect - everyone else saw color. Reshare map and they see color.

A player with darkvision could see no other player tokens unless a torch was turned on. GM could see them as light green when looking through their eyes and would not clearly see them either. Not sure which of the "blackout" workarounds above fixed this one.

If I had a token with lighting - like an NPC swarm of bats - active (blue light) - and removed it out of combat tracker blackouts happened (most consistent way to get them after end turn).

Removing a combat tracker entry and replacing it with another combat tracker entry controled by player the token was lost completely. Sometimes. Never found the pattern for this and it was the hardest to workaround when it happened. Setting Party vision and movement - which we did not want - seemed to make this a little better even though the player was the owner of the DB combat tracker entry.

The lag started gradually and only with one player. Then another - then eventually all of us. This was not normal explainable network lag in my option it was something else. I now understand all the lag complaints which I did not before (because last week I didn't have any with the same code and maps - only after this latest TEST server update).

I don't know what happened or how this is being tested. But unless the developers are running their tests with multiple players over the network (not local) and running decent sized maps with a moderate amount of lighting in them and on the tokens via effects - then that is not a test and you won't see any of this.

I can do this locally all day and never be hit by this stuff as a GM. This needs to be tested in a big map with real LOS in it like pits and lights etc. I had a max of 4 light sources going at any time in all of this.

Was most frustrating time I've ever had with this game.

[kevininrussia]

Something similar was reported here:

https://www.fantasygrounds.com/forums/showthread.php?67514-Poor-Performance-After-Yesterday-s-Update

[SilentRuin]

Something similar was reported here:

https://www.fantasygrounds.com/forums/showthread.php?67514-Poor-Performance-After-Yesterday-s-Update

He did a much better job. Good resource stats - what he specifically used. I was too exhausted after this - plus I looked at the logs. Console log for 3 hours was hardly anything and nothing of note. Network log was like 2 lines. And a cynic could easily say - you used extensions - you used a biggish map with 1 fixed light source and 2 combat tracker torch effect light sources and a lot of columns casting shadows and blocking LOS - the lag was the network...

I have to run a test on monday that should be even tougher - is there a way to set logging in test that would show what I'm seeing for that one?

[Moon Wizard]

As usual, we'll need more information, such as the specific campaign files (zipped up with file share link) as well as details on how to open the images to what you were seeing.

Regards
JPG

[SilentRuin]

As usual, we'll need more information, such as the specific campaign files (zipped up with file share link) as well as details on how to open the images to what you were seeing.

Regards
JPG

You'd have to have all the extensions, modules, and remote players to go along with that test so they could gradually build up whatever we were building up (FOW?) till it killed us dead with lag. And that would still not show the blackout issues without running the players through things as they were seemingly random (though I'm sure there was a pattern somewhere).

This is not something I think can be tested by simply doing a few things to it - needs to be hammered like a real game with players doing locked token movement with lighting - with NPCs with active tokens seeing in various visions as you delete them from combat tracker and map and all the other things I mentioned.

I mentioned all the things that can be done by remote players that caused these issues - remember the GM was last to be effected - instantly receive attack roll - players took forever to receive it. I can never see this stuff with GM only or my local test player. I suspect there is something building up on client sides that is killing things and FOW is my number one suspicion.

I'll check with you tomorrow if you still want things that I don't think by themselves will help you but you for sure can try what I'm suggesting as doing those tests with remote players should get you those results. I found polymorphism extensions switching out a PC with an NPC owned by the player without movement option in FGU set pretty much guaranteed the blackout loss of token would be duplicatable. But plenty see that without that - just more randomly. And never at GM or local player test.

[SilentRuin]

As usual, we'll need more information, such as the specific campaign files (zipped up with file share link) as well as details on how to open the images to what you were seeing.

Regards
JPG

Well I tried to export a module of all my maps and the lighting - loaded them into brand new campaign with one pre gen character loaded in and no extensions - copied the images - unloaded the mod - brought up the images. All the lighting was stripped out of them.

So that's another issue - if you can't export lighting with a map that seems kind of a serious flaw. I'll have to recreate the fixed lighting - fortunately that is one light for this demo - still... I find it hard to believe this is not easily duplicated without a special step by step thing if you have remote clients (5) and run around anything. But I will try to simplify a test. How I will do (blackouts, loss of types of light, can't see other tokens, lag) this when it only happens with remote players? I'm not really sure. But I'll give it a shot.

[SilentRuin]

As usual, we'll need more information, such as the specific campaign files (zipped up with file share link) as well as details on how to open the images to what you were seeing.

Regards
JPG

How do I get this campaign zip to you? Note it has 3 maps we used and I had to put in lights manually as the export failed to transfer any lighting. I only loaded SRD stuff. I put in a character with a torch effect in one map - and an NPC alcolyte with torch effect - and NPC priest next to fixed lighting. Also put a swarm of bats.

You can attach a remote player and see if you can duplicate it - we have 5 remote players though.

End turn on guy with torch from player side would cause map to go black for one or more players - random.
Clicking on the bats from GM side so it was active and blue vision showing - then deleting it from combat tracker reaked blackout havoc on one or more players.
Running the five players through the initial cave map and then into the larger underground ruins (where I set up all the combat tracker stuff right now) eventually killed us all with lag and blackouts as I described earlier. We reached the map I have them in after about an hour and a half due to all the issues we had in the first map but pretty sure you can duplicate this in any map with anything if you have enough remote players and play around enough. But I'm ready to give you mine.

Also we had extensions and other modules as I said - but as I don't think those had anything to do with anything I'm giving it to you in pure FGU. Maps were created with art subscription so you'll need those to see them I suspect.

When you give me a place to drop it I will. If you run it locally as GM or as local player I doubt you'll duplicate anything FYI.

[LordEntrails]

How do I get this campaign zip to you?
The usual suggestion is to upload it to a cloud share service (Google, Drop Box, etc) and then either post a link here or PM it to Moon with a reference to this thread, or open a support ticket with the link and a reference to this thread.

[SilentRuin]

The usual suggestion is to upload it to a cloud share service (Google, Drop Box, etc) and then either post a link here or PM it to Moon with a reference to this thread, or open a support ticket with the link and a reference to this thread.

Is there an email I can just send the zip too for support?

[SilentRuin]

Also while I can duplicate NONE of the things we see in a locally attached player - I can duplicate the total LOSS of the token for a player.

Bring up your local player and assign 2 characters to them. You can get in a state where your looking through the eyes of one character with a torch (with other one in the dark) and not see that character on player side. As it should be. Then switch to portrait of the other character - and nothing. You can't see token you can't find token you can't use token. I absolutely saw this happen during our game with some shared NPCs where they could never get back to seeing a token. The only way I could get back was to reactivate the other token I could still see - then have GM get the other token activated - then switch portrait.

My guess being - activating portrait is not triggering the activation for that token somehow when its not currently visible do to perspective of other token. This could be a more basic problem where things that activate tokens are not doing it when lighting says they are not visible from the last perspective or something. All OWNED tokens (CT entries) of the player should ALWAYS be visible on their map. ALWAYS. And with lighting I don't think that is true in some situations. That is what is causing some of these blackout issues I believe - lights have some how messed with the activation on the player sides and are being lost (as in things that should keep them visible on their screens when they own them - no longer are). Thus causing all sorts of situations where token states or visibility from other players somehow get applied to the owning player and they can no longer operate their map token (gone). Which also means nothing can be seen from their perspective. This is NOT the cause of all blackouts as a lot of the time they can still see "some" of the tokens on the map - just no light. But total blackout would be caused by this.

[LordEntrails]

Is there an email I can just send the zip too for support?
Yea, I think its [email protected], but I'm not sure. Plus be aware that email has file size limits.

[SilentRuin]

Yea, I think its [email protected], but I'm not sure. Plus be aware that email has file size limits.

Sent there with a reference to thread and issues. Again though, can't duplicate it without remote players (as in not local player connected or the GM - it all works with that - except what I noted about controlling multiple players or a shared NPC). But it was consistent and horrible in blackouts, total blackouts (loss of token in map for player), color loss, lag, etc. - everything I mentioned above.

And as a clarification - works always in local player and GM even with all the extensions and modules I used. Which this zip has none of. Hope you can solve it because this is unplayable in games lasting more than 15 minutes in TEST server with lighting active and is worse with movement and party vision "off" which is what this zip campaign has.

[SilentRuin]

I think I have the Rosetta stone of how to screw things up so they start doing all the many horrors - including lag - of last night. But in order to get to happen reliably I have to use polymorphism extension (delete PC CT entry and replace with NPC CT entry owned by player) and even then it does not always happen until it does - then it always happens - which I've not figured out how I've gotten in that state yet. If I do, I'll add that information. I know the other people seeing the lag issue and blackout issues aren't running a druid turning into a bear and even we saw it when nothing with this extension was around - but if I can figure out how this can duplicate that consistently - maybe it can be used to figure out what has happened to the token memory in the engine. Because the state can sometimes (not always) be fixed by resharing the screen or bringing the client back up - I know its not the DB data being changed because it can be repaired without changing any of that. If I can isolate what it takes to guarantee it screws up I can give SW something that can put it in the bad state and they can see why its doing it.

SUCCESS

It seems I can never get it to happen switching between druid and bear UNTIL I execute a move of the map token with map locking on. After I accept it at the GM side. The map will go black completely on player side and as they cannot click on the token (its gone also) they cannot get into a viable view state to see anything. And ALSO a right mouse on the GM map will act like it still needs to accept the move - which I just accepted. And I CANNOT get out of this state in the GM map UNTIL I close the map and reopen it. And once the radial wheel is not longer locked in cancel/accept move I can share it back to player and they are working again.

I can duplicate this 100%.

Also after I recover I move again - accept move again - and the player side MOVES but the GM side of the token DOES NOT. But as I move the window on the player side - just drag the window to new position? I see the GM side slowly move a few ticks to the moved to position - then freeze - then move some more anytime I do something on player side.

Yeah I know its truly bizarre. But 100% duplicatable. And if I close the player while in this state - I can click on token in GM (only app open now) and each time I click on it it moves another tick toward that final accepted movement from a while ago!

[Arghun]

If it can help I'm running your poly extension and one of my players had exactly the same issue last FRIDAY. Switching back to her original form she could no longer see anything. Map was all back for her. And this was with tocken locking ON (GM approved moves) but on the LIVE version (haven't played with the Test version yet).

I'm running the latest poly extension.

[Arghun]

If it can help I'm running your poly extension and one of my players had exactly the same issue last FRIDAY. Switching back to her original form she could no longer see anything. Map was all back for her. And this was with tocken locking ON (GM approved moves) but on the LIVE version (haven't played with the Test version yet).

I'm running the latest poly extension and combat groups, map parcel and equipped effects if that can help

[SilentRuin]

If it can help I'm running your poly extension and one of my players had exactly the same issue last FRIDAY. Switching back to her original form she could no longer see anything. Map was all back for her. And this was with tocken locking ON (GM approved moves) but on the LIVE version (haven't played with the Test version yet).

I'm running the latest poly extension and combat groups, map parcel and equipped effects if that can help

Ever since updates last week. No idea what has changed but switching with poly and move locking movement triggers it for sure.

[Arghun]

I'm interested if you can find a workaround ;-)

[SilentRuin]

I'm interested if you can find a workaround ;-)

Doubt it. But tomorrow I’ll see if I can duplicate doing the things I do in pure FGU.

[Moon Wizard]

Are you making sure that tokeninstance.setOwner is being called when you switch out the token in your extension?

JPG

[SilentRuin]

Are you making sure that tokeninstance.setOwner is being called when you switch out the token in your extension?

JPG

Nope - I do exactly what you guys do in replaceCombatantToken in CoreRPG\scripts\manager_combat.lua --- which is NOT that.

Are you saying that is the issue with all the issues?

[SilentRuin]

Are you making sure that tokeninstance.setOwner is being called when you switch out the token in your extension?

JPG

Also I decided to do what I do manually in FGU without extension to see what would happen. So I deleted the druid I had logged in as a player out of combat tracker - and then shared the alcolyte by dragging it to druid portrait. Then I had to turn movement and party vision on to get it to move - at that point the LAG was there when I moved and got worse and worse each time I moved and accepted.

Granted its not the horrors I get - but close enough. I would for sure not center in on extensions. I'm sure they can aggravate whatever is happening but its not them.

Though if there are new ownership requirements that must be applied to tokens I'd recommend having them reapplied every time a player selects a portrait so they can recover missing blackout tokens.

[SilentRuin]

Are you making sure that tokeninstance.setOwner is being called when you switch out the token in your extension?

JPG

On the off chance you were onto something that nothing else does I added that in to force the setOwner of the owner of the CT - which should already be done under the hood I'd think or nothing would ever have worked - and... no effect. Same bugs, blackout, lag, etc.

Keep in mind I can duplicate this in PURE FGU - the lag part - simply by sharing the alcolyte in that TEST LIGHT campaign I mailed you guys by dropping it on player portrait and trying to move with movement locked. Slow - slower - unplayable in three easy moves.

[SilentRuin]

Hopefully SW will look at not just the RAW FGU example I provided to duplicate the lag - but also the ability to cause blackouts with an extension that has worked until the middle of last week. Well, still works mostly - but given the things I described earlier. Something that can force the blackout problem without the randomness that people in raw FGU hit would be worth looking at.

[SilentRuin]

My goal today (since I already gave them a way to 100% duplicate the killer lag in 3 moves) is to figure out a code workaround for the blackout for my test game tonight. Right now I'm down to NEVER USE MAP TOKEN LOCKING - or you will die of lag. And also Polymorphism will blackout the player screen - which if you have movement locking off - will be fixable by GM resharing the map (or maybe player closing and opening it from images themselves). Which may let me test tonight. I've warned the pests (that's what I call players - they live to be exterminated for the greater harmony and peace of my NPCs) that if we see problems like we did Saturday in first 10 min we are going to bail.

Also, noticed that in one older project I have that was converted from LIVE - polymorphism will bring up the NPC sheet no problem and bring up the PC sheet when it switches back. But in my TEST LIGHT campaign I see the sheet bringing up the NPC no longer doing it. NO idea why - but figured I'd mention it if it gives some sort of clue to what is going on.

[SilentRuin]

For SW based on Moon Wizard talks...

100% lag duplication demo.

1) Use TEST LIGHT campaign I provided as is. No extensions nothing just bring it up. Leave all defaults as they are in everything except turn Movement and Party Vision ON.
2) Bring up combat tracker in GM.
3) Bring up underground ruins map that is expanded out so you see the 3 light sources (all CT stuff is already placed in there).
4) Bring up test player to attach to TEST LIGHT.
5) Choose Zoloft the one PC I provide on player side.
7) GM shares underground ruins map to player.
8) Drag CT entry link for Alcolyte to player portrait in upper left to share to player. Acolyte sheet should pop up on both player and GM screens.
9) On player side select Alcolyte and drag token in map some distance.
10) Approve movement on GM side.
11) You will see lag.
12) Move Alcolyte on player side again some distance.
13) Approve movement on GM side.
14) You will see horrific lag.
15) Move Alcolyte on player side again some distance.
16) Approve movement on GM side.
17) You will see unplayable lag.


100% blackout duplication demo.

1) Use TEST LIGHT 2 campaign I provided as is. Attach only polymorphism.ext. Leave all the defaults as they are - DO NOT CHANGE ANY OF THEM.
2) Bring up combat tracker in GM.
3) Bring up underground ruins map that is expanded out so you see the 3 light sources on GM (all CT stuff is already placed in there).
4) Bring up Eltoora's sheet from CT entry on GM.
5) Go to actions tab on Eltoora's sheet on GM.
6) Click the ACTIVE button at top sub window on GM.
7) You see sheet change - token change to B (bear token) - and CT change on GM.
8) Click the ACTIVE button on the new bear sheet on GM.
9) You see the sheet change back - token change to B (bear token) - and CT change on GM.
10) You can move around the token in either state. That is how it should work on GM and Player.
11) Bring up test player to attach to TEST LIGHT 2.
12) Bring up Eltoora as your PC on player side.
13) Bring up combat tracker on player side.
14) GM shares underground ruins map to player.
15) Go to actions tab on Eltoora's sheet on player.
16) Click the ACTIVE button at top sub window on player. Serious lag I've never seen before sometimes at this point - you have to wait here till you see the new sheet pop up. Weird.
17) You should see sheet change - token change to B (bear token) - and CT change on player. DO NOT TOUCH THE TOKEN.
18) Click the ACTIVE button on the new bear sheet on player.
19) You see the sheet change back - token change to B (bear token) - and CT change on player.
20) Now from player side - move the token (locking is turned off so it will move with no lag issues).
21) Go to actions tab on Eltoora's sheet on player.
22) Click the ACTIVE button at top sub window on player.

BLACKOUT

Token is gone and only resharing the screen will bring it back. Unless you had movement token locking on then your in for a world of hurt. Which is why its off in this demo.

Once TEST LIGHT 2 is messed up I see weird stuff. But starting from the zip I give you for every test will get you to the same point.

[Sterno]

Just curious, SilentRuin... do you see the same lag if token movement is unlocked?

[SilentRuin]

Just curious, SilentRuin... do you see the same lag if token movement is unlocked?

I can't duplicate as instantly. But I see it in other places, like if I change druid to bear in my TEST LIGHT 2 blackout duplication campaign I gave them with polymorphism.ext - the change from player side seems to take forever sometimes. Never seen that before. Usually instant.

That has communications between GM and host though so if client side message queue is messing up that is not a shocker. Otherwise, Not really tried to much till today building these tests with token locking off in my lighting tests so don't know.

[SilentRuin]

WOOT! TEST LIGHT 2 Blackout demo issue was found and fixed and will be in next release. Unofficially :)

Not sure if that fixes all the types of things I saw and can never duplicate but thank you SW for the effort!

[SilentRuin]

Testing of my FGCON16 game tonight with my last map was night and day difference from my Saturday session testing this thread describes. We had no issues.

But... I turned off all map token locking movements.

I made VERY sure we did not delete anything out of combat tracker - with or without lighting - it all just stayed as tombstones when they died.

All the polymorph effects were on creatures the GM controlled.

This 3rd map played nicely without map token locking on.

No shared NPCs.

Fairly open map - no closed in spaces.

[houseofdexter]

Maybe a bit of light on this. We have a laptop in our test group that has HDD that is very very slow. On this computer he gets extreme lag with Light Sources and Dynamic Lighting, when we turn off lighting everything works fine, this was from Last Thursday.

[lostsanityreturned]

Maybe a bit of light on this. We have a laptop in our test group that has HDD that is very very slow. On this computer he gets extreme lag with Light Sources and Dynamic Lighting, when we turn off lighting everything works fine, this was from Last Thursday.

It is a scaling issue sadly, players in my group with stronger computers were brought to their knees by 3 light sources. Weaker computers had unusably low fps with 1 light source.

It took 4 tokens with lights to get me down to single digit framerate -laughs-

[houseofdexter]

In our setup, there were probably 20 light sources added to the map. We had one light source effect on one of the players, this computer didn't have any problems with light and fps. The one player that was having problems (and also had the slow hdd) had devilsight.

[Moon Wizard]

Closing this thread. Just pushed a new build.

If you are still seeing performance issues, please submit the steps to recreate seeing this issue, along with a copy of your campaign data.

Regards,
JPG