Big Useability issue wit 2.7.4.0

[CampbellR66]

Just had to abandon the 3rd of the last 4 game sessions. Since the 2.7.4.0 upgrade we are suffering from random disconnects across the DM and 5 players, its got to the point that this is unuseable.
We cant see any internet issues
skype still runs fine
The PC's are mixed spec from good to average.
lots of file transfer errors on clients

We have been running for nearly 2 years without many issues. In fact the previous version was very stable...only the upgrade seems to have changed things.

Held of logging for a couple of weeks to see if it settled down or a fix issued....


Help before our group dies....

[Moon Wizard]

Campbell,

I'm sorry to hear about the network issues. There have been 3-4 games where this has cropped up according to what I'm seeing on the forums. It's either not happening to others (or at least only minimally).

I'm currently working with Primarch on the Test version (v2.7.5) to try and track down some of the connection issues.

If you want to roll back to 2.7.2, the installer in the downloads section is still v2.7.2. Also, if you are running a Lite, Full or Ultimate license, you can set your version by using the Dev mode of the Test Mode Manager tool (available on the downloads page).

Current Versions via the Test Mode Manager
Release = v2.7.4
Test = v2.7.5 (beta)
Dev = v2.7.2

If you have any ideas or details, I would appreciate anything you have. You can see more discussion on this in the Laboratory thread for v2.7.5.
Thread Link (https://www.fantasygrounds.com/forums/showthread.php?t=14088)

Thanks,
JPG

[CampbellR66]

Thank you for the response

I will contemplate a rollback to 2.7.2, will there be any issues with the modules and campaign files that have been used with 2.7.4

The only additional observations we have seen is players logging in seem to bump other players out. Lower spec machines seem to be more vunerable.

The file transfer errors relate to attempts to reload parts of the campaign that are very old... has something happened to cause clients to re-cache everything even though it is already in the cache.

We had one session last week that seemed to be normal with only one drop out... no logic to this, as i mentioned before we have been running for a couple of years with the same hardware and campaign.

love the product so really want to sort it out

Regards

Rob

[Moon Wizard]

I wonder if it has something to do with age/size of campaign; and how much data is transferred on login.

* How big is the campaign db.xml file on the host machine?

If you are running a 4E game, you can try using the /flushdb chat command to reset the shared data in the campaign. In long running games, this can drastically reduce transfers on login as old campaign data is no longer shared automatically.

* Can you post some of the file transfer errors you are seeing?

I added some code to output messages on network errors, and it uncovered a minor issue with portrait tokens. I want to see if that is what you are seeing, or if it's something new/different.

Thanks,
JPG

[CampbellR66]

Hi

We are runing JPG_d20 ruleset

the db.xml is 2,169 kb

Are there log files on the client i could ask my playesrs to send? I dont see any errors on the server end?

Other wise i will ask them to copy some down

[Moon Wizard]

The errors are not currently tracked in the log file, though they are in the Test version (v2.7.5).

The campaign is a good size, but not too big. For 4E, mine is 1.8M, and I only keep one adventure at a time.

If your group is open to it, you can move to 2.7.5 until you get the drops, then back-rev to v2.7.2 to keep playing.

As long as the ruleset files stay the same, then the FG version changes should not affect the campaign database, since the ruleset controls the database structure.

Regards,
JPG

[CampbellR66]

I will load the test tool and see how easy it is before i try to get the less technology confident players to give it a try.

The module file that we are using is 35,046 Kb which is where most of the content is stored. A slow conversion of the Mysteries of the Moonsea FG campaign module.

[CampbellR66]

Where do we find the log file?


Client or server ...or both ?

PS ...week 5 still sux ...5 players getting booted every 5 mins

[CampbellR66]

More info

All players reporting loads of file transfer problems...really old files.

Seems to fail when i display a map. 4 - 5 players booted within seconds..

Why is it retransfering all these old files?

[orcishdoug]

Yet again we have upgraded the program and it has fallen over, we get in, load up and get booted out this is getting very annoying. Someone please help

[CampbellR66]

Update ... 10pm ...group giving up ...week 5 washed out.

We even tried deleting the campaign and module .dat files on all the PC's

Still no improvement.

If we use test utility and select dev button will this roll back a version?

[Moon Wizard]

It depends on which files are generating the errors, maps/images or tokens.

Any time that you have shared a map/image with a player, that map/image will be rechecked every time the client loads. It should only re-transfer the files if the copy of the file in the cache does not match the copy on the server.

Here are 2 options to resolve a caching issue with maps/images:
* I have attached an extension which will add the /flushdb chat command to the d20_JPG ruleset, to allow you to reset all shared database items while running as the host.
* There have been situations where the cache file gets corrupted, and deleting the cache on the clients helps. (i.e. delete the cache subfolder under FG2 data directory) You can try this, and let the client re-download.

For tokens, all shared tokens are sent at the beginning of every client connection, since they are not cached. Token caching is already in the works for v2.8. Also, if a token is used anywhere in the campaign and it is shared with a user, that user will request the token when they get access to that map/image/tokencontrol. (i.e. either on login or when shared initially)

A few questions:
* Which version did you end up running on last time? (v2.7.2, v2.7.4 or v2.7.5)
* Are the clients seeing the error messages in the chat log or somewhere else?

If you are running v2.7.5, you can send me the console.log files that contain more data. It is located in the FG2 data directory, and is reset every time you start FG. Matching host and client files are best, but I'll take anything.

The FG2 data directory can be found under the FG2 group on your Start Menu.

Thanks,
JPG

[CampbellR66]

Thanks for responding ..... the players have stayed on still

The errors are in chat box and are mostly to do with portraits but this might be becaues we deleted the campaign.dat file.

We are back to 2.7.2 using test mode manager.

Seems to have sorted it out ...some issues from clearing cache but no dropping characters in 30 mins

[Moon Wizard]

The portrait errors are from using tokens created by dragging portraits from upper left onto map. When the map loads, if the character whose portrait is used which is also not logged in, then this error is shown. This error was just silently ignored prior to v2.7, and is one of the fixes coming in v2.7.5.

Regards,
JPG

[CampbellR66]

Is this portrait on map issue causing players to get disconnected?

I can see a lot of our old maps have had portrait tokens left on them from old game sessions...some of these characters are dead and no longer used and some have been fully deleted, this may be the part of the problem.

Can you provide more information on what determines that a cached file needs to be refreshed, The module.dat file i assume re-caches each time i re-export it and open it in the main campaign? or does it track individual files within it? What is the definition of the change.

None of the errors seen so far seem to be with files in the module, but are instead items that should be in the campaign.dat file. The old content in this has not changed on our workstations in months.... just new items added?

once agin thank you for responding and helping us out. There are a few posts from players in the campaign in this thread and another ... all related to the same issues.

Regards

Rob

[Moon Wizard]

Based on my recent review of the networking code (which I inherited from previous developers), I don't see any reason why a failed file transfer would kill the network link. The file transfers are implemented as a separate socket request to the host. However, I did not write the code, so I may be missing possible non-obvious interactions.

v2.7.5 resolves the portrait token errors by correctly recognizing when a token was created from a portrait vs. loaded from the tokens folder. If a portrait token is missing, it will create a blank token instead, and request the portrait from the host. (i.e. no error)

As far as I am aware, this behavior has not changed since before I took over the code. I just added the error messages in v2.7 to get more information on potential network issues. I reviewed the changes in the code from v2.7.2 to v2.7.4, and the only changes were log file format and some LUA changes.

If your group is running Lite/Full versions, I would appreciate if the host and one of the clients could run the Test version and check it out. I ran it on Wednesday without any issues, but I also do not use portrait tokens in my game.

Regards,
JPG

[CampbellR66]

I have all the players set up with the test mode manager so we will run next thursdays game on the 2.7.5 test version and see what happens.

I am running some LAN based test on 2 of my own pcs to see what happens

We finnished last Thursday session all back on 2.7.2 with clients havng cleared the local cache. This allways causes problems in t own right a itcauss lots of file transfr errors and players unable to complete the initial ruleset download before character selection. So it is difficult to see if it has fixed things...but we did get 5 clients connected and stable for 30mins and displayed some maps without anyone getting booted.

One poor guy was however stuck in an endless loop trying to load the initial data prior to character selection...it kept failing and forcing a complete retry.

Regards Rob

[CampbellR66]

Based on my recent review of the networking code (which I inherited from previous developers), I don't see any reason why a failed file transfer would kill the network link. The file transfers are implemented as a separate socket request to the host. However, I did not write the code, so I may be missing possible non-obvious interactions.

v2.7.5 resolves the portrait token errors by correctly recognizing when a token was created from a portrait vs. loaded from the tokens folder. If a portrait token is missing, it will create a blank token instead, and request the portrait from the host. (i.e. no error)

As far as I am aware, this behavior has not changed since before I took over the code. I just added the error messages in v2.7 to get more information on potential network issues. I reviewed the changes in the code from v2.7.2 to v2.7.4, and the only changes were log file format and some LUA changes.

If your group is running Lite/Full versions, I would appreciate if the host and one of the clients could run the Test version and check it out. I ran it on Wednesday without any issues, but I also do not use portrait tokens in my game.

Regards,
JPG

I have a console log file showing a disconnect error but it will not let me attach to this post ...invalid file type.

Probably need to allow .log files?

[ddavison]

I added log to the allowable extensions type. Please try to attach now and let me know if you have any problems.

[CampbellR66]

Log files from client and server

[Moon Wizard]

Sent you a private message to get more information.

Thanks,
JPG

[CampbellR66]

Moon

I ran the tests again today and the errors reappeared.

Log files attached and they were produced on client an server in parallel...
Logs show 3 disconects by client where server takes 2-5 minutes to spot the client is gone and clear down the connection.

I repeated the cycle 3 times .... just did random dice rolling on client until the chat log stops responding ...disconnect follows a minute or so later.

[Sakusammakko]

Following this thread with interest.

Since moving to 2.7.4, one player in my group has been getting booted at least once an hour. She had never dropped a session before. Have ignored the problem as a minor inconvenience.

If there are more out there with this problem, I will pay attention. May follow Moon Wizard's advice above.

[Moon Wizard]

Thanks for the logs.

A couple questions I came up with:
* Is either the client and/or the server on a wireless connection?
* What kind of router do you use? (brand and model number)

Thanks,
JPG

[CampbellR66]

Both are LAN connected via a netgear 200mb powerline network.

The router is a netgear WNR2000Cable modem/router

Trying to get some internet connected clients on for test today if some of my players are willing.



Thanks for the logs.

A couple questions I came up with:
* Is either the client and/or the server on a wireless connection?
* What kind of router do you use? (brand and model number)

Thanks,
JPG

[CampbellR66]

One of the regular players took part in the test (The Druid Xylia)

Running an average workstation on broadband

The system generated lots of nice drop out errors straight away.

logs attached

There are 2 client logs as there was one complete FGII crash to the desktop with the 'Let microsoft diagnose the problem@ type dialogues.

No server crash

[primarch]

Hi!

It be interesting to see if everyone experiencing this problem has a netgear router.

I also have a netgear (WNDR3700 Rangemax Dualband Wireless N Gigabit Router).

Is anyone having this issue have a router other than a Netgear?

This could be valuable info for Moon Wizard if it proves to be the case.

Primarch

[CampbellR66]

I have had this router for about a year ..long before the problem started.

players have a mix of router types



Hi!

It be interesting to see if everyone experiencing this problem has a netgear router.

I also have a netgear (WNDR3700 Rangemax Dualband Wireless N Gigabit Router).

Is anyone having this issue have a router other than a Netgear?

This could be valuable info for Moon Wizard if it proves to be the case.

Primarch

[Griogre]

Interesting idea. It could be a firmware update. Given CampbellR66's lack of trouble before - Do the netgear routers auto-update?

[primarch]

Interesting idea. It could be a firmware update. Given CampbellR66's lack of trouble before - Do the netgear routers auto-update?

Hi!

Yes. Netgear routers have auto update that kicks in when you access the router.

There was a firmware update this last Sunday (after my game), but I don't remember when the last one was nor when the last one before that occurred and its time relation with the updates 2.7.3/2.7.4.

It may be nothing, but bears watching. Perhaps a firmware update not playing nice with the changes in those versions?

This Sunday I will play using 2.7.2 to confirm Campbell's finding that the behavior does not occur under that version and only under latter ones. If the trend holds, then it will be easier for Moon to narrow down the cause.

Primarch

[Moon Wizard]

Here are my plans and research so far:

* My next step is to revert the changes between 2.7.2 and 2.7.4 to see if that addresses the situation. Since they were log file format and LUA changes, it doesn't seem like they should affect the network connections. However, it's worth a try.

* I double-checked the libraries used to build FG, but nothing has changed there since 2.6.5.

* I'm researching various reasons why network sockets might break, which is where the questions about wireless and routers come from. There are a lot of people online that have had problems with broken network connections across a variety of apps, but p2p apps seem to be affected the most. (like FG)

* I'm also reviewing ways to enhance FG's network implementation to better handle errors. (i.e. attempt reconnect vs. exit game) Due to the nature of the original network implementation, this may not be a quick fix.

Also, I want to thank everyone who is helping out on this.

Cheers,
JPG

[CampbellR66]

Our game group meets again this evening ... we will see how we get on

I will get logs from as many players as possible and send

[CampbellR66]

Ran the game session for about an hour before we gave up. Same 2 errors

1) Random client disconnects - takes server a while to clear connections
2) Mass player boot by server - takes players a long while to detect they are booted.

5 players
3 client logs and one server log attached
2 player lost logs by accidently restarting the program and erasing file.


Um ... cant upload some of the files (server log is 4,000kb+) in fact only one of the 4 files is under the file size limit for log file attachment.

ahh a zip is allowed .... attached zip


Our game group is pretty much suspended pending a fix or a full roll back. 5 weeks on the plot has kind of gone cold, so dont want to re-start until its fairly certain its going to work.

Will stick with helping test for a bit if it adds value.

[CampbellR66]

Forgot I also had a LAN based local test client running. Log attached
This client took a full 30 minutes to work out the server had booted it. I had to roll some dice to force it ...i expect it would still be showing connected if i had left it.

Also spun up a rolemaster rulset game ... same issues

Re zipped full file set and attached

[Moon Wizard]

I thought that you were able to run the game using v2.7.2, and that you were using v2.7.5 primarily to test out changes to see if we could resolve. Has that changed?

If so, then there might be a more general issue with something breaking connections on your network. There are many topics on the Internet in general about certain routers, firewalls or other network situations that can cause sporadic connections. Given that there are only 2 people using FG that I am aware of that have repeatable broken connection issues, I'm inclined to think that is the case here.

However, I do agree that FG should handle broken connections better. (i.e. reconnect options, broken link detection, etc.) This is one of the underlying assumptions of how TCP and the Internet should work. Unfortunately, these things are not simple changes, and would potentially introduce more issues, which is why I have not attempted to implement in v2.7.5 yet. Also, I inherited this networking code and just begun to investigate it, so my concern is that too much fiddling could break other things. In fact, I might need to change network libraries completely, which is a large change.

At this point, I'm hoping that we can figure out what might be causing broken connections in your environment to address short term, either running v2.7.2 or finding the cause of the sporadic connections on your network.

Thanks,
JPG

[primarch]

I thought that you were able to run the game using v2.7.2, and that you were using v2.7.5 primarily to test out changes to see if we could resolve. Has that changed?

If so, then there might be a more general issue with something breaking connections on your network. There are many topics on the Internet in general about certain routers, firewalls or other network situations that can cause sporadic connections. Given that there are only 2 people using FG that I am aware of that have repeatable broken connection issues, I'm inclined to think that is the case here.

However, I do agree that FG should handle broken connections better. (i.e. reconnect options, broken link detection, etc.) This is one of the underlying assumptions of how TCP and the Internet should work. Unfortunately, these things are not simple changes, and would potentially introduce more issues, which is why I have not attempted to implement in v2.7.5 yet. Also, I inherited this networking code and just begun to investigate it, so my concern is that too much fiddling could break other things. In fact, I might need to change network libraries completely, which is a large change.

At this point, I'm hoping that we can figure out what might be causing broken connections in your environment to address short term, either running v2.7.2 or finding the cause of the sporadic connections on your network.

Thanks,
JPG

Hi!

As time goes by, I am more and more inclined to agree with you and think it is a router related issue.

I will report my findings in full after I run my two games to fully test my hunches, but suffice to say that the more I read about my particular router and proceed to tweak it and fully understand its in's and out's the more I realize that even trivial omissions may have a big impact.

My preliminary findings is that under 2.7.2 I do NOT see the behavior, but I will put that to the real test on Sunday.

Of course, now, I must see if the elimination of the behavior was due to the rollback or due to my careful step by step configuration.

The way I have setup my test run will give me an answer one way or another, since Sunday I will run the game using 2.7.2 and on Monday I will run the game using 2.7.5.

If Sunday's game runs without incident and the behavior is manifest on Monday's game then the problem is very likely to be something within the updates since 2.7.2 that causes the behavior.

If the issue is absent on both days, then it was a router issue and I will share my thoughts on what I changed and what was the causal issue.

Fingers crossed that I can bring resolution to the issue. :D

Primarch

[CampbellR66]

If no more logs or stuff is needed we will go back to 2.7.2 and see if it all goes back to normal.

I still get the same issues when running the server and client on the LAN .... can an issue with the router cause problems with this?

[Moon Wizard]

From what I have been reading online, it looks like routers, firewalls and wireless connections are the primary culprits for interrupted socket connections.

I think the reasons vary a bit, but the general ideas I saw online were either related to timeouts when the connection is quiet or shut down due to type or amount of traffic on the connection.

Wireless issues are a different beast, but you are using a wired connection, so not an issue.

Cheers,
JPG

[primarch]

From what I have been reading online, it looks like routers, firewalls and wireless connections are the primary culprits for interrupted socket connections.

I think the reasons vary a bit, but the general ideas I saw online were either related to timeouts when the connection is quiet or shut down due to type or amount of traffic on the connection.

Wireless issues are a different beast, but you are using a wired connection, so not an issue.

Cheers,
JPG

Hi!

Moon, I have been trying to track down some sort parameter that deals with timeout on the router, but have found none. I know there is a time out parameter somewhere, perhaps related to the TCP protocol accessible through the cmd prompt?

I know I have seen some routers have a configurable time out parameter, but the one I have doesn't have one. I wonder if it can be manually configured some other way.

Any ideas are appreciated.

Primarch

[Moon Wizard]

It may not be something that is configurable, or might be a side effect of another setting.

Basically, for any network troubleshooting, I strip out layers until I was out of layers or the problem resolved.

* Turn off Windows firewall
* Turn off any antivirus or spyware blocker apps
* Turn off router firewall/SPI
* Review router port forwarding
* Plug computer directly to Internet

Obviously, I would not recommend leaving all the network security disabled, but it usually helps narrow the cause to the layer that is the issue.

Other random bits of information that might be helpful

I haven't updated my router firmware in about 3 years, since it's working. However, just browsing through my router's manual (D-Link DGL-4100), I found a few settings that I would look at if my connections were breaking often.
* PPPoE connection: Maximum idle time
* PPPoE connection: Reconnect mode
* DHCP: Lease time
* Lots of port forwarding options (schedule, trigger ports, inbound ports, ...)

Here's an article on conflicting firewalls on Netgear site:
https://kb.netgear.com/app/answers/detail/a_id/1111

Interesting post from MS to disable IPv6 in Windows 7 to resolve an issue:
Post (https://answers.microsoft.com/en-us/windows/forum/windows_7-networking/windows-7-drops-my-internet-connection-and-resets/258e32e6-8bf2-457d-ae85-3161e6e0741c)

Regards,
JPG

[primarch]

Hi!

Hehe, funny you mention the IPv6 thing. I read the same thing earlier this week and disabled it.

I also found the PPPoE settings and switched it to "always on" instead of "dial on demand" although I could have easily increased the timeout number as well.

I suspect that once all the testing is done it will turn out to be the router settings, but I'll let my Sunday and Monday games with the parameters previously described settle the issue. :)

In any event I think its important everyone learn the ins and outs of their particular routers and never assume that stock settings are "port forwarding" friendly. Sometimes you got to tweak it just right. ;)

Primarch

[primarch]

Hi!

Update on new thread here: https://www.fantasygrounds.com/forums/showthread.php?t=14288

Primarch

[CampbellR66]

Same group of players on the same hardware and networks

5 players all reverted to 2.7.2

All pretty much back to normal we had one mass player drop where everyone got booted as the initial logins took place.

Beyond this one player got disconnected once during a 3 hour session.

Server crashed once, but i am putting this down to the Java update installer popping up ...it crashed just after that.

This level of disruption i would consider normal .... but importantly we got to play a game session.
:D

[Zeus]

I ran a 4E game last night under 2.7.4 with 5 players (1 Observer). 3 of the players were in the UK, one was in France and the last observer was connected in from the US. I also had a Skype conference running with all six of us using VoIP without issues.

With the exception of 1 small hiccup where one player was disconnected (the player in France) the game ran without any issues. I don't put this down to FGII though as this particular player is always having network problems due to the quality of his connection (cheap broadband). Ironically, there was no delay in waiting for FGII to 'release' his connection. He simply disconnected and then immediately re-connected.

There were a couple of times where one or two of the players seemed to lock up for a minute or two, but in both instances their FGII clients eventually unlocked (within 90 secs or so) and play continue unaffected. Again I put this down to local background processing interfering with their FGII clients and not down to the FGII software itself.

In summary over the last 3-4 weeks I have not seen any detrimental issues with 2.7.4, certainly nothing that would suggest to me that it has introduced any net issues beyond those present in 2.7.2 and 2.7.3.

[CampbellR66]

Got to the bottom of our stability and disconnect problem..

It was due to a wireless card getting re-enabled on the server. Once this second card was disabled again everything settled down running over both LAN and WAN.

We are now back on the current release and all are happy again.

I think the card re-enabled when i updated some drivers and stuff ..... always need to be carefull making the "Nothings changed..." Statement.

Thank you to everyone who looked into this...

A bit embarassing but best to Fess up to these things so the knowledge is recorded

[Moon Wizard]

I appreciate the extra response effort. On the forums, it's common for issues to go dark as they resolve, without feedback on the final result.

Thanks,
JPG

[NW_Salmon_Watcher]

Hi, Moon_Wizard.

Here is one more to ponder...

My players are both getting regular file transfer errors (please see attached screenshot).

These errors happen everytime my players Join Campaign, however there is about a 2-5 minute or so lag (it seems to vary) between when the campaign starts to load and when this error arises. During that lag period, the players cannot easily open anything, including their characters, tokens, etc.

I saw your February 24 post about a "flushDB" extension, which I downloaded, but no details on where to go from there, i.e, to which folder I save it and when I should run it (after players log in, before, after errors occur, etc.).

Can you provide some more suggestions please, as this error - while not fatal like the one CampbellR66 posted - is slowing down our game session starts, and being parents of small kids, our gaming sessions are already too short as is! Losing 10-15 minutes per log in (especially if we have to restart for any reason) is a headache we'd like to see ended.

THANKS so much!

[Moon Wizard]

If you are getting lots of file transfer errors, I would try deleting the FG2 data directory cache on those client machines which are getting the errors. This will remove any issues due to any cache corruption. It also forces a redownload of the ruleset and campaign files.

For the flushdb extension, it simply resets all "viewing" status of data in the database. You just type /flushdb after you load the campaign plus extension. It's more of a one shot thing.

Please redownload the flushdb.ext file from earlier in this thread. I just updated with a version that does not clear owners, only viewers.

Regards,
JPG

[NW_Salmon_Watcher]

Thanks, Moon Wizard.

Will heed your advice and try both remedies.