4.4 is crazy slow to display local images in reference manual builder

[Syldar]

Hello, I updated to the latest 4.4 version, and noted that now that when I edit a reference manual with the builder and load a page with an image (or load an image in the campaign image folder), even locally on a 7gb/sec read/write SSD (M1 Max), there is a huge latency. I takes 4 to 6 secondes to load. This is crazy. It was not like that with previous version. I tried it both with a previously edited campaign and a new one I'm currently creating with 4.4 version.

Please fix this asap, as it is very very annoying. And completely not normal on a fast SSD. So I imagine what it would be when playing with friends online…

Thanks for the support.
Nico

[Zacchaeus]

I'm not seeing this. And I haven't heard any other reports.

What ruleset are you using? Are you using any extensions? Does this happen in a new campaign without extensions? What are the file sizes and formats of the image that you are using? Any other relevant information would be useful and maybe also zip up a sample campaign where you are seeing the issue.

[Syldar]

The ruleset I used in WOT20E, with or without extensions, it does not change anything. I tried with existing campaign and a new one, it is the same. The files are jpg and png files, around 300kb to 1mb files. And I have around 20 image files. But when I open a campaign and open a huge 17mb map from the image database, it loads a bit faster than before (which is weird). The slowing seems to only occur in the reference manual builder.

[Trenloe]

Which ruleset is WOT20E?

[Syldar]

A vampire ruleset for world of darkness 20th anniversary, available here: https://www.fantasygrounds.com/forums/showthread.php?72799-World-of-Darkness-20th-Anniversary-Edition-Ruleset-(Revive)-for-FGU/page7&highlight=wod20e

[Trenloe]

Can you reproduce the slow issue and then compile your logs and attach them here. Details on compiling the logs can be found in the Wiki here: https://fantasygroundsunity.atlassian.net/wiki/spaces/FGCP/pages/1242136781/How+to+Compile+Logs

This will help us to determine exactly what you're running and if there are any issues reported in the logs.

[Moon Wizard]

Also, a 17MB map is way beyond the recommended specifications, and will cause slowdowns in many situations. Given the changes we had to do to support webm/webp, it could be having a larger effect now.

Try shrinking your map to under 4Kx4K resolution for best performance. You can try larger, but nothing larger than 8Kx8K.

Regards,
JPG

[Syldar]

OK, I got my log. I opened the reference manual I'm building and went to see with a local image on one page. Same slow loading happens when I open the campaign folder in the assets. Crazy slow to list / refresh the list of files (with only a few images of max 1mb in it).
You can ignore the last line of log, as this is an error that has nothing to do with the problem. The ruleset don't show the compile log icon, so, in order to create my log, I used a bug that is still present in the ruleset to get the log to open.
But, again, the error have nothing to do with the problem (it was there before, still there now) and it is only in version 4.4 of FantasyGrounds that it happens.

I have no idea on how to upload a file on the forum, so I used mega to host it.
The logs are here: https://mega.nz/file/IXwRhRJA
Password: 8iuCUOWaIPJII0qoHr79PmQ7qyrTNRC3NoLSRjOcmRk

Hope that helps.

[Syldar]

I hear you, but the problem don't occur with this particular image. This image is not even present in the campaign I have the problem with.
To give more detail about that 17mb jpg file, it is 6250 x 4250 pixels, in RGB 8 bits, without guides.

Also, what would happen if I would reduce the resolution of the map? I created all walls, line of sight, lights, would they be automatically scaled? I guess not… If not, how to do it without redoing all the hard work (if there is a way)?

[Moon Wizard]

There is nothing in the logs that will give us much information about performance issues; as the logs tend to track what you have loaded and whether there are any errors. Lots of errors, low GPU memory or lots of extensions can sometimes point to considerations that could affect performance, but I don't see those particular items here.

For the image reduction, that seems very large for a 6Kx4K file; but I guess it depends on quality and compression ratios used for the JPG. If you resized the map, you would need to stretch the new smaller resized one larger to match the LoS using the image data panel Layers section.

Questions
* Do you get the slow downs when you disable all extensions?
* Do you get the slow downs when using the map in a CoreRPG or 5E campaign?
* Do you get the slow down if you remove that map from the FG data folder completely?
* Do you have a copy of the image file you can share to look at?

Regards,
JPG

[Trenloe]

There is nothing in the logs that will give us much information about performance issues; as the logs tend to track what you have loaded and whether there are any errors. Lots of errors, low GPU memory or lots of extensions can sometimes point to considerations that could affect performance, but I don't see those particular items here.
I asked for the logs to get an idea of which version of the WOD2E ruleset was being used and which extensions were being loaded; also, to see if there were any warnings.

I've tried to recreate the issue with the same settings - ruleset, extensions and 150% scaleui - but don't see any slowdown when viewing pages with images in the refman builder. But, I'm running on Windows, not on a Mac.

@Syldar - sorry this is proving hard to nail down. Would you be able to put a short video together, showing the steps you follow and the slow loading of images please?

[Syldar]

* Do you get the slow downs when you disable all extensions?
Yes, I get slow downs when I disable all extensions. For example, I just need to open the asset window, go into campaign folder, it takes around 15 second to open the folder. (I checked for permissions repair, everything is fine).

* Do you get the slow downs when using the map in a CoreRPG or 5E campaign?
Yes, same huge slow down with a CoreRPG campaign I created without extensions, only a few images (no huge images, just to be sure). So the problem seems to be CoreRPG.
No, no slow down in a 5E campaign.

* Do you get the slow down if you remove that map from the FG data folder completely?
Yes, the slow down is still there, the image have nothing to do with the slow down, as I expected.

* Do you have a copy of the image file you can share to look at?
Yes, but I guess I can't upload it here. I can't upload it from the forum system, I can select the file, but then it does not upload, don't show any button to click ok, or showing upload progress.
Here it is on Mega:
https://mega.nz/file/4PoliDyD
Password: LV30fZAcc8sFFHaRxWNSEgNxMaqTg_5mKcNtzX3dALM

So, bottomline, the problem seem to be caused by CoreRPG, because creating a test campaign with CoreRPG (no special ruleset, only CoreRPG), no extension at all and just a few simple image, try to open assets then campaign folder, it takes forever to open. That is the problem from my point of view. At least on Mac version.
By the way, when will there be a native Apple Silicon version of FantasyGrounds, that would be amazing.

[seycyrus]

Why oh Why, is that map almost 17 MB?

Just looking at the resolution, I am confused.

[Trenloe]

@Syldar: Would you be able to put a short video together, showing the steps you follow and the slow loading of images please?

[Syldar]

Haha, anyway, that is not the problem. It could be saved to the same resolution, but with compression.
I think the most important here is to understand why the problem is happening with any image (not this single special image). CoreRPG seems to be the culprit here (at least on Mac).

[Moon Wizard]

I tried the map file on both my Windows desktop machine, as well as my MacBook Pro 2019. For the Windows machine, the image came right up in both the Assets window and when creating a new image record with the asset. For my Mac laptop, the image took about .25 seconds to open in Assets; and maybe .5 seconds for the full image in a new image record.

I'm not sure why your machine is loading slowly at this point. CoreRPG doesn't have any impact on image usage, as that is core to the application itself. My guess is that there is more going on in terms of data load on your machine, in terms of the number and size of image files that you have loaded into the FG data folders.

Can you try these steps?
* Open the FG data folder via the button in the upper left of the FG launch screen.
* Rename the images/tokens subfolders in the FG data folder.
* Start FG, and create a brand new CoreRPG campaign.
* Open the Library->Assets window.
* Do you see any slowdown?

* Next, in the Library->Assets window, click on the Images filter button at the top, then click on the Folder button at the bottom.
* Add the single "Underground Castle 01.jpg" file to that folder that was just opened.
* In the FG Assets window, click the Refresh button next to the folder button.
* Go to the Data bag within the Assets window to view the thumbnail for the castle.
* Do you see any slowdown?

Basically, we need to isolate what is causing any slowdowns to the minimum amount of information needed to see the issue, so we can figure out what might be causing the issue.

Also, I tried opening and re-exporting the JPG file in Photoshop, and saw significant reductions in size.
Full quality JPG - 10MB
Very High quality JPG - 6MB
Full quality WEBP - 13MB
High quality WEBP - 1.5MB
I've attached the High Quality WEBP to this post.
Note: We are planning to recommend High quality WEBP for all submitted content to are store/forge.

Regards,
JPG

[Syldar]

thanks, I'll try this.
But anyway, on Mac (M1 Max with 4tb SSD 7,5gb/sec, 64gb of memory), after deleting all campaign and tokens, starting a new campaign with CoreRPG (no extensions at all) and a few BASIC SMALL IMAGES (20 images, maximum 2mb), the slow down is huge (around 12 seconds) when I open the campaign folder from the asset window. Something is clearly not normal here. And it is with CoreRPG, nothing else (after doing this elimination work).

At least on Mac. Which is really frustrating, when you know how fast these machines are, especially in terms of SSD (even with Rosetta ; can't wait to see a full native apple silicon version of FantasyGrounds…).

Could you check with the people responsible for the development of CoreRPG what is this issue (at least on Mac)?

[Moon Wizard]

I am one of the developers. I consolidate the features from the main application developer as well as the ruleset developers, and work with them to develop features, make builds and answer developer questions. As I mentioned above, I tested with your image on both my Windows and Mac machine without any slow down issues at all.

From your post, it sounds like you did not limit your testing to the single file you provided, and I'm concerned that you have not disabled/removed all of the other data files to narrow the testing.

As a developer, we do not have any special power to know what exactly is going on for your particular hardware or operating system configuration; so, we suggest scenarios that minimize the number of possible factors so we can figure out what factor is triggering the issue you are having. Unless we can identify the issue, we have no idea how to fix the problem for you.

At this point, it might be best to try uninstalling, renaming the current FG data folder, and reinstalling from scratch with a brand new, clean data folder with no custom images.
Then, add a single file at a time to a brand new campaign folder until you see the slow down, and let us know when you see the slow down. (i.e. when refreshing the Assets window after adding the file, when searching in the Assets window, when creating a map, when moving the map, etc.) Also, we would need details of the exact steps you are following, so that we can try to recreate here.

Another alternative is to potentially zip up your entire FG data folder and place in a file share, along with instructions for which campaign to open, which exact windows to open, etc. (i.e. step by step walkthrough). If you decided to take this approach, this will take more time due to the large amount of data needing to be coordinated between developers to try to recreate the issue.

Another thing you can try (if you have another machine) is to see if the same FG data folder has the same issues on another machine.

Regards,
JPG

[Syldar]

Hmm, that seems very long and tedious… Can I upload the campaign and the ruleset in a zip file?

Question, what happens if I delete files in a campaign, like db.session.20230619 ? Would I lost data entries?

[damned]

db.session files are backups - one is made each session. FG only reads from db.xml

[Syldar]

OK, I found the problem. It was the fact that I had some images in JPG, but in Grayscale. The problem was not the size of images, only that many images were jpg files in grayscale 8 bits.
This is what was slowing down things. I'm so happy that this problem is solved.

Thanks a lot for the help, guys. Maybe you should check that on your side, comparing two campaign copies, one with RGB jpg files, one with Grayscale jpg files, and compare loading time. On my side, it was day and night!

[Nylanfs]

Wouldn't grayscale maps be LESS overhead than RGB?

[Syldar]

Yes, they would, reason why I used grayscale images at first for images that were anyway in grayscale. But, turns out it seems very bad for the system, so my grayscale images are now in RGB and everything is fine. Quite odd, I agree, but that's how it is so far…

[Griogre]

Thanks for investigating, that seems like an odd bug. You might send FG support one of those grayscale images and reference this thread so they can investigate further. It's possible that even though the image was grayscale the resolution was still very high as FG just cares about resolution not actual colors. Grayscale would help on compression file size but not FG LOS calculations which are Height x Width X Color depth in bits.

[pindercarl]

Yes, they would, reason why I used grayscale images at first for images that were anyway in grayscale. But, turns out it seems very bad for the system, so my grayscale images are now in RGB and everything is fine. Quite odd, I agree, but that's how it is so far…

Thanks for tracking down your issue. Any images that are not in RGB or RGBA color have to be converted. I'll consider logging a warning for palettized images.

[Syldar]

They were basic images in JPG format, grayscale, 8 bits, saved from Photoshop, with compression 12 (best quality), and their weight were from 41kb to 1,15mb, so nothing crazy here. Grayscale, was really the problem here, for sure; as soon as I changed all my grayscale pictures to RGB in all my campaigns, everything went back to crazy fast.