Auto-Saving Keeps Getting Longer - What to Do?

[seansps]

Hello everyone,

I am hoping a dev can help me figure out what is going wrong here. This is the second time I've created my campaign from scratch (new campaign, reload modules, import characters.) I really don't want to do that again, but the auto-save every 5 minutes keeps getting longer and longer and I don't know what I am doing wrong. I am keeping all my images and extra story entries in a separate module, and trying not to add bloat, but it seems like just standard use of FGU increases the save time, despite the db.xml file only being about 1 MB.

Right now my save time is at 7-10 seconds and it is infuriating trying to run the game when it keeps locking up. The party just completed the final boss of Princes of the Apocalypse, but they want to follow my homebrew hooks to get to level 20, so, I hope I can get this solved!

Here is a link to my campaign:
https://drive.google.com/file/d/1B7dACYIKajRLftjGdH3eYePl6xKIEaff/view?usp=sharing

Any help much appreciated however, I would *highly* appreciate the ability to OPT OUT of Autosave event, even if you have to put a disclaimer like "DO THIS AT YOUR OWN RISK OF LOSS OF FILES!" If it only saved automatically on quit of FGU, I would be so much happier. Just me 2 cp.

Update (1/19 4PM ET): It seems I have a solution for now. The fix was to revert changes I made to one map in my campaign (Player version of Plunging Torrents) and then the save time when back down to under 1s. This seems to indicate a bug to me, however, because the only changes I really made to this map were drawing on it (and placing tokens of course.). I think the issue/bug is with drawing and undoing, because I consistently had issues with the pen where it wouldn’t work, and then using the eraser made the pen drawing appear. I’ve reported that issue before separately, but no fix yet.

[Zarestia]

I don't see this on my computer, save time is according to the console 0.1 seconds, could be a MAC things though.
What is your OS and FGU version?

[seansps]

I don't see this on my computer, save time is according to the console 0.1 seconds, could be a MAC things though.
What is your OS and FGU version?

Weird! Yes I wonder if it is due to MacOS? I am on macOS 12.1 (Monterey). And the latest version of FGU (4.1.13).

51069

[Sterno]

I see it on my PC with when the campaign gets large (5mb+ DB file). Big thread about it here: https://www.fantasygrounds.com/forums/showthread.php?70661-Seemingly-random-soft-locks-while-creating-large-modules

Your db.xml looks to only be about 1 MB though. Rough.

[seansps]

I see it on my PC with when the campaign gets large (5mb+ DB file). Big thread about it here: https://www.fantasygrounds.com/forums/showthread.php?70661-Seemingly-random-soft-locks-while-creating-large-modules

That's the thing, though. It's not, mine is only 1 MB. I also have an older version that I use for making additional modules (almost the same size) and it saves in 0.3s.

Also, I just tested this in Windows, and it is happening there too for me. Takes 7-10s to save there as well.

[Sterno]

@seansps Something I saw was that if I was actively working in the campaign, autosave could take a long time, but if I just let the app idle, autosave was quick. Are you seeing the long save times even if it's just idling?

You can check by leaving the console open (/console in the chat window to open it) and just walking away for 15-20 minutes. You should see "Campaign Saved" messages with the time it took to save appear in the console.

[seansps]

@seansps Something I saw was that if I was actively working in the campaign, autosave could take a long time, but if I just let the app idle, autosave was quick. Are you seeing the long save times even if it's just idling?

You can check by leaving the console open (/console in the chat window to open it) and just walking away for 15-20 minutes. You should see "Campaign Saved" messages with the time it took to save appear in the console.

Sadly, yes… I see it when idling or doing the /save command. :(

[nephranka]

I am with @Sterno in that this has been mainly a DB size issue but given your description it could be something corrupted in the DB file hanging up the process? I have noticed that a Windows export takes 5-10x longer than a Linux export of the same campaign. So, clearly there are differences in the way the code interacts with the HD and file system.

[seansps]

I am with @Sterno in that this has been mainly a DB size issue but given your description it could be something corrupted in the DB file hanging up the process? I have noticed that a Windows export takes 5-10x longer than a Linux export of the same campaign. So, clearly there are differences in the way the code interacts with the HD and file system.

I guess that would be indicative of a bug then. Why would it get corrupt with normal use? It’s a pretty new campaign, too. I am very dismayed with the performance of FGU lately from all sorts of angles, and the frustration with this auto save feature is really trying my patience as a customer, to be honest.

[Sterno]

Your autosave issue does seem to be different. Hope you get it sorted out!

[seansps]

Your autosave issue does seem to be different. Hope you get it sorted out!

Thank you, me too!! If I can’t resolve this I may just switch to Foundry and deal with the lack of automation. But I am hoping I don’t have to.

[Zacchaeus]

I tested and it seems to take about 6 seconds to save on my computer - which is extordinarily slow.

I did notice that you have every module under the sun open and when I closed them all except PotA this cut 2 seconds off the save time. The modulestate.xml file notes many more modules that I don't own so you might try closing those as well. I note also that you are using a number of hefty extensions (from the extensionstate.xml file) and maybe closing those off and testing in a new campaign would help you test whether those are affecting the postion. Your spellcasting characters have all sorts of wierd notations next to their spells so I don't know where that comes from - probably an extension maybe? That too might be causing issues. Otherwise I'm not seeing anything which might indicate why it's so slow. My own campaign in PF2 with no extensions and six characters at level 10 saves in 0.1 seconds and my db.xml is currently 3Mb.

[seansps]

I tested and it seems to take about 6 seconds to save on my computer - which is extordinarily slow.

I did notice that you have every module under the sun open and when I closed them all except PotA this cut 2 seconds off the save time. The modulestate.xml file notes many more modules that I don't own so you might try closing those as well. I note also that you are using a number of hefty extensions (from the extensionstate.xml file) and maybe closing those off and testing in a new campaign would help you test whether those are affecting the postion. Your spellcasting characters have all sorts of wierd notations next to their spells so I don't know where that comes from - probably an extension maybe? That too might be causing issues. Otherwise I'm not seeing anything which might indicate why it's so slow. My own campaign in PF2 with no extensions and six characters at level 10 saves in 0.1 seconds and my db.xml is currently 3Mb.

Thanks for looking into it!

Yes I use a lot of modules because I reference and use a lot of content. I hope and wish that is not the problem here.

The spell notations are from the module for Rob2e’s spell effects. Never had an issue with that before.

I suppose it *could* be one of the extension or modules, but this does not explain why when I first set it up a month ago it was saving at under 1s and slowly crept up to 8s.

Something else is going on here. Thanks for looking into it though, but I don’t see a solution for me here.

[Nylanfs]

Just a thought, you don't have your live campaign data on a shared folder like a Google Drive or anything do you?

[seansps]

Just a thought, you don't have your live campaign data on a shared folder like a Google Drive or anything do you?

Nope- it’s just saving to the default campaigns folder in FG, not a OneDrive/iCloud/GDrive folder.

[nephranka]

I tested and it seems to take about 6 seconds to save on my computer - which is extordinarily slow.

I did notice that you have every module under the sun open and when I closed them all except PotA this cut 2 seconds off the save time. The modulestate.xml file notes many more modules that I don't own so you might try closing those as well. I note also that you are using a number of hefty extensions (from the extensionstate.xml file) and maybe closing those off and testing in a new campaign would help you test whether those are affecting the postion. Your spellcasting characters have all sorts of wierd notations next to their spells so I don't know where that comes from - probably an extension maybe? That too might be causing issues. Otherwise I'm not seeing anything which might indicate why it's so slow. My own campaign in PF2 with no extensions and six characters at level 10 saves in 0.1 seconds and my db.xml is currently 3Mb.

Just a point about the DB size. It is both the size of the DB and the timing of events while auto saving occurs, that seems to be some of the source of issues for the process, at least in my experience. Once I get to 5 MB it is really untenable so I have developed ways to reduce the overall size of the DB to help.

[seansps]

Just a point about the DB size. It is both the size of the DB and the timing of events while the auto saving it occurring that seems to be some of the source of issues for the process, at least in my experience. Once I get to 5 MB it is really untenable so I have developed ways to reduce the overall size of the DB to help.

Same. Again, in my case here, it is only 1 MB. So what is the save process really doing?

I feel like we should be given a use-at-your-own-risk disable-autosave feature until they can optimize this and fix it.

[Zacchaeus]

Thanks for looking into it!

Yes I use a lot of modules because I reference and use a lot of content. I hope and wish that is not the problem here.

The spell notations are from the module for Rob2e’s spell effects. Never had an issue with that before.

I suppose it *could* be one of the extension or modules, but this does not explain why when I first set it up a month ago it was saving at under 1s and slowly crept up to 8s.

Something else is going on here. Thanks for looking into it though, but I don’t see a solution for me here.

As your campaign progresses more code gets added to the db.xml and so it gets bigger and could take longer to save. Extensions add code to the db.xml and if you remove an extension then it can leave code in the db.xml hanging and it may casue recusrion if the save is trying to find stuff that no longer exists. Having every module open also makes the lists in the campaign huge; for example you have 93 pages of items. Now I don't know how FGU decides what to save but suppose that it needs to run through all of those lists to see if something needs to be saved. If it does then that's a lot of pages to work through. You can still access all your material but do so when you prep. If you want to use something from Fizban or Volo or Mordenkainen then open them when you prep and copy whatever you want into the campaign and then close the module. As I said closing off all the open modules that I could shaved 1/3 off the time it took to save the db. So having 30 modules open absolutely has an effect on the save time. You also don't really need the player versions of modules open; nor modules which contain only images or tokens.
Looking at the extensions list that you use there's a couple of odd things like 878507-Linkfill - I don't know what that is but if it's messing with links then that is one of the biggest things that need to be processed. Another thing to make sure of is that if you are using an extension that you have the latest version. It's always possible that some error is casuing a problem that you're not even aware of.

Overall, your campaign isn't that big; you haven't got a massive amount of images shared. I did notice you had a sizeable file in the moduledb for PotA most of which seems to have been changes made to various maps but I honestly don't know how that affects things. They have to be saved but I don't know if that is part of the save process or not. But given all that; we have to focus in of what might be the issue and elimiante them as possibilities. So closing modules and disabling un-needed extensions (better - starting a new campaign without the unwanted extensions) to see if that helps resolve the issue.

[seansps]

As your campaign progresses more code gets added to the db.xml and so it gets bigger and could take longer to save. Extensions add code to the db.xml and if you remove an extension then it can leave code in the db.xml hanging and it may casue recusrion if the save is trying to find stuff that no longer exists. Having every module open also makes the lists in the campaign huge; for example you have 93 pages of items. Now I don't know how FGU decides what to save but suppose that it needs to run through all of those lists to see if something needs to be saved. If it does then that's a lot of pages to work through. You can still access all your material but do so when you prep. If you want to use something from Fizban or Volo or Mordenkainen then open them when you prep and copy whatever you want into the campaign and then close the module. As I said closing off all the open modules that I could shaved 1/3 off the time it took to save the db. So having 30 modules open absolutely has an effect on the save time. You also don't really need the player versions of modules open; nor modules which contain only images or tokens.
Looking at the extensions list that you use there's a couple of odd things like 878507-Linkfill - I don't know what that is but if it's messing with links then that is one of the biggest things that need to be processed. Another thing to make sure of is that if you are using an extension that you have the latest version. It's always possible that some error is casuing a problem that you're not even aware of.

Overall, your campaign isn't that big; you haven't got a massive amount of images shared. I did notice you had a sizeable file in the moduledb for PotA most of which seems to have been changes made to various maps but I honestly don't know how that affects things. They have to be saved but I don't know if that is part of the save process or not. But given all that; we have to focus in of what might be the issue and elimiante them as possibilities. So closing modules and disabling un-needed extensions (better - starting a new campaign without the unwanted extensions) to see if that helps resolve the issue.

As mentioned before:

“I suppose it *could* be one of the extension or modules, but this does not explain why when I first set it up a month ago it was saving at under 1s and slowly crept up to 8s.”

Same modules. Same extensions.

At this point I feel like I need to jump through seven hoops to get FG working optimally.

I’ll be ok with the long save if it only happens on quit or when I request it. Refusing to offer a disable of autosave is weird. I’m kind of at my wits end.

[Trenloe]

I feel like we should be given a use-at-your-own-risk disable-autosave feature until they can optimize this and fix it.
That's a good idea.

I had a look at the campaign as well - running on Windows 10 on a fairly good spec computer, /save in chat would take between 8 and 9 seconds. I reverted changes on one module (D&D Princes of the Apocalypse) and the save time reduced to 0.2 seconds. Obviously that's not a fix, as it removes any of the modifications made to that module - based off the changed data for that module it appears to be changes to some images that make up the bulk of the data saved.

Investigating further, if I revert changes on only "The-Plunging-Torrents-Player" map, that reduces the save time to what I'd expect (0.2 seconds).

Thanks for providing the test campaign. Hopefully this will help the devs investigate further as we know it's an issue with one specific changed image in a module.

[seansps]

That's a good idea.

I had a look at the campaign as well - running on Windows 10 on a fairly good spec computer, /save in chat would take between 8 and 9 seconds. I reverted changes on one module (D&D Princes of the Apocalypse) and the save time reduced to 0.2 seconds. Obviously that's not a fix, as it removes any of the modifications made to that module - based off the changed data for that module it appears to be changes to some images that make up the bulk of the data saved.

Investigating further, if I revert changes on only "The-Plunging-Torrents-Player" map, that reduces the save time to what I'd expect (0.2 seconds).

Thanks for providing the test campaign. Hopefully this will help the devs investigate further as we know it's an issue with one specific changed image in a module.

Thank you!! This is what I was looking for. I will try that. That is very weird. Why would my map changes cause it??

I have a theory now: I was drawing a lot during the final battle (to show water levels and such) and using the pen and eraser and undo… and pointers. That’s when the lag started. I suspect the drawing is bugged.

[seansps]

AGHH! Sure enough, reverting the changes to that ONE map caused it.

I suspect it was all the drawing I did. Now saving is back to 0.1s. WOW! @Trenloe THANK YOU!

[Trenloe]

AGHH! Sure enough, reverting the changes to that ONE map caused it.

I suspect it was all the drawing I did. Now saving is back to 0.1s. WOW! @Trenloe THANK YOU!
That's great news! Hopefully this will give the devs a good example of where to look for the data issue. Thanks so much for bearing with it and providing detailed information of your issues!

[Moon Wizard]

Do you happen to have a copy of the campaign (or the moduledb folder in the campaign) prior to reverting that map?

Regards,
JPG

[seansps]

Do you happen to have a copy of the campaign (or the moduledb folder in the campaign) prior to reverting that map?

Regards,
JPG

Yep! The google drive link on the original post hast it.

Simply reverting the player map for “Plunging Torrents” fixed the save issue.

[damned]

I am with @Sterno in that this has been mainly a DB size issue but given your description it could be something corrupted in the DB file hanging up the process? I have noticed that a Windows export takes 5-10x longer than a Linux export of the same campaign. So, clearly there are differences in the way the code interacts with the HD and file system.

Generally applications hand things like this off to the operating system.
They dont rewrite them for different operating systems.

[nephranka]

Generally applications hand things like this off to the operating system.
They dont rewrite them for different operating systems.

Thanks for the clarification. I have notice the difference for a long time but never reported it. It is strange to have such a large difference.

[Moon Wizard]

It looks like this issue is specifically related to token FoW data save times. I'm going to include a fix in the next client release to speed up the save times on this type of data; and I've filed a ticket for Carl to review this scenario to ensure that FoW data is being calculated correctly when he has a moment.

With the fix that I am working on, the save times on my machine go from 6-6.5 seconds to 0.1 seconds again.

I'm hoping to target a new build for next week; but it depends on ongoing development timing.

Regards,
JPG

[seansps]

It looks like this issue is specifically related to token FoW data save times. I'm going to include a fix in the next client release to speed up the save times on this type of data; and I've filed a ticket for Carl to review this scenario to ensure that FoW data is being calculated correctly when he has a moment.

With the fix that I am working on, the save times on my machine go from 6-6.5 seconds to 0.1 seconds again.

I'm hoping to target a new build for next week; but it depends on ongoing development timing.

Regards,
JPG

Amazing! Thanks, Moon!

[Trenloe]

It looks like this issue is specifically related to token FoW data save times. I'm going to include a fix in the next client release to speed up the save times on this type of data; and I've filed a ticket for Carl to review this scenario to ensure that FoW data is being calculated correctly when he has a moment.

With the fix that I am working on, the save times on my machine go from 6-6.5 seconds to 0.1 seconds again.

I'm hoping to target a new build for next week; but it depends on ongoing development timing.

Regards,
JPG
Nice one! :)