FG Browser with search [CoreRPG, 5e, SWD, other]

[sirnoobsauce]

This extension adds a browser-style functionality for the purposes of organizing and searching records and reference library data.

I made it for my own purposes as I was having a hard time managing all of the campaign records in a long-running game with a lot of recurring NPCs and other campaign records.

I also wanted a search feature to be able to perform full-text search of all records and library data in a single place, because I would often forget where something was and wanted an easier way to find NPCs based on data in their notes, find story records based on their content, or search the library reference data.

The example screenshots below are taken from the 5e ruleset just because that is a very popular game system, but the extension should work with any ruleset based on CoreRPG. I myself use it in my own homebrew ruleset.

Installation

This is available over on the Forge: FG Browser (with search) (https://forge.fantasygrounds.com/shop/items/1133/view)

Or if you prefer, you can manually download the ext file here (https://github.com/bakermd86/FoogleBrowser/releases/latest) and install in your Extensions directory.

Usage

It can be used in 2 ways, as a campaign record type, or as dynamic browser windows that will not be stored permanently.

To create a persisted browser type campaign record, it is available under the "Browsers" campaign record type on the sidebar. You can create a blank browser and add records either by drag/dropping them into the tab list, or by using the search page to look up records. Here is an example:

https://raw.githubusercontent.com/bakermd86/FoogleBrowser/v1.0/doc/5e_manual_example.gif

When using the search page, you can use the filter fields to filter the results by record class, name or the source of the record. You can either left-click on search results to open them in a new tab and immediately switch to the tab, or middle-click to open them in a new tab without switching to it (like most web browsers). Tabs can be closed via the middle mouse button.

Tabs can also be renamed either by double-clicking the tab, or right-clicking and using the edit option in the radial menu.

You can also use the browser without linking it to a campaign record. To do so, just use the chat command:


/foogle <search string>

The optional search string parameter, if provided, will perform a search automatically when opened.

If you have a dynamic browser window that you want to persist to the campaign data, you can right-click and there is a save option in the radial menu which will copy the dynamic browser to a persisted one. Here is an example of a dynamic browser:


https://raw.githubusercontent.com/bakermd86/FoogleBrowser/v1.0/doc/5e_usage_example.gif

Update 1.1.0

Several people reached out to me to ask about improvements to the search functionality (mainly search performance). The initial version of the search was almost an afterthought, so it was pretty basic. The new version adds a search indexer that indexes the campaign and module records on campaign load, so that searches will have better performance. It also implements search scoring to try to sort the results so that the most relevant results are at the top.

The "Module Indexing" option button can be used to select which modules to index for search, and the "Rebuild Index" button can be used to trigger the indexer to run again if something gets messed up in the search index (or if "Index on-load" is disabled).

Update 1.2.1

The search indexing worked ok on very fast computers, but introduced issues in some environments (it was particularly sensitive to read speed on the drive holding the module data). To that end, in 1.2.1 I implemented a background event loop to handle the indexing (since the FG Lua environment doesn't support coroutines).

In order to work well across the widest possible range of environments, the indexing behavior can be modified to suit your needs:

https://github.com/bakermd86/FoogleBrowser/raw/master/doc/settings.png

To keep things as simple as possible, there are some built-in config profiles accessible via the slash command:


/indexMode <min|low|normal|high|max>

If you only want to use the browser for tabbing records, and don't care at all about search, /indexMode min will disable indexing on-load entirely, and indexing will only run when you manually click the "Rebuild index" button. So if you want to have the browsers for organizing records, and don't care about search at all, then do that.

The others just gradually turn up the indexing workload and scheduling priority.

The main setting that is going to impact the responsiveness of the client UI is the "Event Loop Priority" setting. Lower values of this setting will make the indexing take longer, but will return control to the FG client loop more frequently, and so will make the client feel more responsive while indexing is running. If you notice stuttering or other performance issue with the UI while indexing, you can set this lower. If you want indexing to run faster, and the UI is sufficiently performant for your liking, you can turn it up.

There is also a blocking setting if you just want indexing to lock up the client and run as fast as it can.

The default setting for priority is "Automatic" which tries to adjust the scheduling priority dynamically every second to ensure that it is returning control to the FG client main thread at least 30 times / second. Automatic should well in most cases, but you can turn it up/down as desired.

You can also turn off indexing for non-text fields (number, image, dice, links, etc.) for a slight increase in performance, and you for formattedtext fields for a modest boost. Basic Indexing restricts indexing to simple string fields only, and uses a much simpler lexical tokenizer for a significant boost in indexing performance at the cost of a less thorough search feature.

[damned]

fantastic sirnoobsauce!

[Zacchaeus]

Jolly well done.

[Jiminimonka]

Nice idea!

[anstett]

Quick question. Does this work on the Player side as well?

I am thinking of notes they make on character sheets, or other extensions that allow the players to to create separate notes (from Mad Nomad for example).

[sirnoobsauce]

Quick question. Does this work on the Player side as well?

I am thinking of notes they make on character sheets, or other extensions that allow the players to to create separate notes (from Mad Nomad for example).

That's a good idea. I went ahead and implemented that. The build is still queued for processing on the Forge, but you can manually get the ext here (https://github.com/bakermd86/FoogleBrowser/releases/tag/v1.0.1) if you want it now (and should be on the forge within a day, whenever it lets me set it as the live version).

[spencerg]

Absolutely fantastic and essential sirnoobsauce - sent you a msg about your indexing implementation - if you need me to test another alpha build, shoot me a message.

[rocketvaultgames]

With today's update, FGU is taking upwards of 8 minutes to load a campaign. It is usually less than 2 minutes.

Autosave and using "/save" are taking 55-75 seconds instead of the usual 1-2.

The problem is the same on (relatively) high-end desktop and laptop.

Return to launcher is taking multiple minutes instead of seconds.

I waited 10 minutes after finally getting a player connected and was still unable to see any characters to select. Usually they pop up after 30 seconds or so.

Disabling this extension resolves the issue.

[spencerg]

I can confirm the above issue - we had to turn it off in our game last night. Something else strange I noticed was random freezes, even when no records had changed and all indexing is set to off. It was acting like it was spiked on memory.

Core rpg - 6 modules loaded - not too much.

Later i’ll start removing one extension at a time and see if that makes a difference:)

[sirnoobsauce]

I pushed a version to the forge with a quick workaround for this while I look into options to make a more robust solution. I will explain more fully what is going on below, but the short version if you just want it to work and are having performance issues like those described above, go into the options and scroll down to the "Search Options" settings, and I would recommend (for now) setting "Persist Campaign Index" to "Off" and then hit the "Rebuilt Search Index" button. That will take a couple of seconds, but that delay when you rebuild the index is how much time the indexing should add when loading or saving the campaign.

I made "Off" the default this setting in the build, so it may automatically pick that up and you might not need to change it. The other options can be used to speed up the indexing if you still feel it is too slow. For people who only want to use the extension for the tabbed record containers and don't care about search, then I would recommend setting Basic Indexing to On, and turning the other four settings off. Then rebuild the index. With that done, the index building should be very fast and the impact on load/save should be minimal.

If you're willing, I would actually appreciate some feedback on how much time they add to building the index, and their impact on how useful the search results are. Because I want to tweak the default settings to give the best experience to people in other use cases. To explain the options:

Basic Indexing - This has 2 effects. First, it simplifies the lexical tokenizer to only index on plain words. If this is off, the tokenizer will perform basic lexing functions like indexing a record for "attack" and "attacking" if it sees "attacking" in a record. Turning this off speeds up indexing by a fair bit. The second effect is to ignore all non-string fields in records. So formattedtext, number, die, token, etc. fields will all be skipped in indexing.
Index Formatted Text - This will just skip formattedtext fields. If basic indexing is on, this doesn't really do anything since formattedtext would be skipped anyways, but I found that lexing the formattedtext fields was the heaviest part of indexing by a fair bit, so this will improve performance by a bit, without being quite as extensive as the basic indexing.
Index Modules - Toggles whether or not to include records from loaded modules
Index Reference Material - Toggles whether or not to include the reference pages from loaded modules

Anytime you change a setting, you can use the "Rebuild Search Index" button on the bottom pane of the options window to re-index and see how long it takes. The indexer will also log in the console.log how long the indexing took (in seconds) in a print like this:


[4/7/2023 6:45:31 AM] s'SearchIndexer.buildIndex: ' | #0.78899999999976

Now as to what was causing the huge performance hits (and the intermittent hangups):

I wanted to avoid having to build the entire index at every startup, so I decided to cache it in the CampaignRegistry, which will accept lua tables. But the index is very large, 30-100MB in my tests, depending on how many records and how large they are. But in most normal cases, the CampainRegistry is on the order of a few KB, it is just a pickled lua table of key/value pairs. So if I store the search index in my test campaign in the CampaignRegistry, the file (CampaignRegistry.lua in the campaign directory) goes from ~2.3KB to 86MB.

This wasn't causing an issue on my normal PC, but for testing I tried to run the extension on a slightly older PC and it became totally unusable, as mattvictim and spencerg described. I am suspecting that either the NVMe m.2 drive or the more-modern CPU in my main PC is just fast enough that it covers up the issues of handling the pickling/unpickling of such a large CampaignRegistry file. But regardless, I found significant performance improvements by just turning off the index caching altogether and getting the CampaignRegistry.lua back to a more normal size. I decided to err in the direction of making the indexing a bit leaner by default for now. So the default behavior now is to use Basic Indexing with Index Modules on, and everything else off.

The massive CampaignRegistry is why saving the campaign takes so long, because the in-memory cached index has to pickled and written to the disk. This delay on saving is also why spencerg was experiencing the intermitted hangups in the UI, as FG will periodically save the campaign while it is running. Anytime that happens, the client will hang for however long it takes to save.

The issues with the client also seem to be caused by the massive CampainRegistry. It looks like the CampaignRegistry is replicated from the host to the client when they connect. When I chuck in 100MB of search index in there, it has a big impact on that operation. I wasn't seeing that in my testing here because the laptop I was using as a client is on the same network as the PC, so that transfer is super fast. But I ran a wireshark capture while connecting and I did see way more data go over the wire when connecting with the cached search index vs connecting without it. So that probably caused the client issue

The reason I looked for a way to cache the index in the first place is due to a bit of a limitation in the scripting system in FG, which is the inability to run code asynchronously. From my testing, no matter what you do, any Lua code execution blocks the main UI thread, and hangs the entire application while it is running. So something that would be better to run as a background task like building a search index has to run synchronously with the UI. That is why I wanted to try to cache the index, but looking into the reports here today make me suspect that is not going to be a good approach.

For now, I will wait to hear back if disabling the caching and tweaking the indexing options gets you into a good state, and then figure out what defaults I should set based on your feedback. If you want to follow up via a DM to let me know how it goes on this version, I would appreciate the feedback.

[sirnoobsauce]

I did a little testing to see (at least in my compute environment) what the impact of the various settings are on this version. The first thing to note is that I am definitely going to remove the index caching, so leave the "Persist Campaign Index" setting off for now until I remove that. Too much potential there to cause issues with slow networks / slow disks (when I moved the campaign data directory to a platter drive... forget about it).

As for the performance of the other options, I did A/B testing for all the settings. The option to exclude reference materials seems to have no meaningful impact on performance in my tests, so I will probably remove that option since it doesn't seem to hurt anything. The other 3 do all have meaningful impacts on the performance. Disabling module indexing will offer the greatest speedup, but of course means that you can't search records from modules. The "Basic Indexing" option also greatly speeds up indexing, at the cost of not being as thorough. And disabling formatted text indexing offers a modest boost to indexing performance.

The times in this table are for a test campaign using 5e with the standard 5e PG and DMG and the 3 SRD modules (Bestiary, magic items, data). Obviously these performance data are specific to my system, so you could have slower/faster indexing on your computer. But theoretically the % change row-to-row should be vaguely similar. Note that this is how long it takes to build the index at campaign load, or when clicking the "Rebuilt search index" button. Modifications to records only have to re-index the modified record, and that only takes milliseconds, regardless of the options:



Basic Indexing
Index Formatted Text
Index Modules
Average Load Time


Off
On
On
3.65s


Off
Off
On
1.75s


On
N/A
On
0.95s


Off
On
Off
0.05s


Off
Off
Off
0.04s


On
N/A
Off
0.02s



Let me know if that is consistent with what you are seeing, or if you are having a different experience with the indexing behavior on the new 1.1.1 version.

[spencerg]

I will definitely test later today. Got your messages. Thanks for your hard work on this!

[Mattkilt]

I loaded this into my system, and when it is enabled none of my games will load at all. it just locks up at the title page after I tell it to load the campaign.

[scoot138]

Spencer was in a campaign where I was running FGBrowser, my CampaignRegistry.lua was at 40mb, i have nvme m2 drives, 32gb ram, i5-9600 @3.7GHz. I was getting intermittent hangup. I unloaded FGbrowser, and all was well. I had toyed around with the settings enough to get the registry to that size. I find that I can go from about 2MB to 500MB of data in the registry depending on what modules I have loaded when I rebuild the index.

Currently set my options to basic index and module index on, the rest off. Its been 4 minutes and still locked up. This was starting out with an empty CampaignRegistry.lua

To be fair, I am testing against A LOT of modules. Prior to this release, it would take a minute or two.

[spencerg]

So far my experience with todays new release has been stable.

It obviously will take a good 3-5 mins to load up the campaign - but I really I dont have any issue with that - the benefits FAR outweigh a slow load-up.

Since updating, I've had zero weird hang up's in the last 2 hours and that ive been hammering it, building maps.

Image of settings attached. I will let you know If I have any issues.

Turning OFF Persist index = slow loadup always, but no stability issues yet that I can tell . search is blazing fast now, even more-so than before.

What I noticed prior to the option to disable it writing to the campaign registry was very odd.

I'd be doing nothing, not changing anything, just reading...or tabbing out too the browser. I'd come back in, and it would be frozen (milky overlay) -- give it a few, and it would unstick.

It was like it was just struggling at random.

Mostly likely because FG is polling and periodically backing things up, and it was chocked on the index data.

Image attached.
56993

[scoot138]

Ran a new search with Basic and modules on the rest off, I disabled all non world/campaign/bestiary information modules and the rebuild took 24 seconds.
I recall, when persist was default enabled, there were modifiers in the CampaignRegistry.lua for what modules were indexed, I liked that, having the ability to set false to the ones I don't want indexed, would be a great feature.
When bringing up a client, it takes an extremely long time to fully load up the background decal, player selection, etc, this being on a client on the same 1gb wired, same switch, internal network, cannot imagine how long it will take across the WAN, with multiple clients pulling data. This occurred regardless of the type of index I created.
In all, this tool is in my mind is the next best thing to sliced bread and butter in a tub, even if there are still bugs.
I am happy to test out anything you need testing performed on.

[sirnoobsauce]

Thanks for all the feedback. I don't understand why my index rebuilds are soo much faster, I have fewer modules loaded, but the difference between 3 seconds and many minutes is odd... I will try to do some testing and see if there is some secondary issue happening.

Either way, I think based on the feedback I will set it to default to basic indexing with no modules, to ensure that the OOB experience won't prevent people from loading their campaigns, and then add an interface to select modules to index. I will take a look at that later today when I have some time.

[Ludd_G]

Hi,

One thing I've just noticed is that with this extension loaded the same campaign was using over 4 gig of extra RAM, pushing my RAM usage into the 93+ per cent, a level which, on my old 8 gig of RAM laptop, would've caused these type of slow downs and freezing. Now with my newer 16 gig of RAM laptop, my RAM usage during a session, with all t'other stuff going on (zoom, edge etc. etc.) is usually in the 60%, at which point enabling this extension pushes it to the mid 90%. This may not be the most recent updated version though, I haven't run 'check for updates' today yet.

Cheers,

Simon

[sirnoobsauce]

Hi,

One thing I've just noticed is that with this extension loaded the same campaign was using over 4 gig of extra RAM, pushing my RAM usage into the 93+ per cent, a level which, on my old 8 gig of RAM laptop, would've caused these type of slow downs and freezing. Now with my newer 16 gig of RAM laptop, my RAM usage during a session, with all t'other stuff going on (zoom, edge etc. etc.) is usually in the 60%, at which point enabling this extension pushes it to the mid 90%. This may not be the most recent updated version though, I haven't run 'check for updates' today yet.

Cheers,

Simon

Yeah I only really tested the indexing on my PC, and I have a ton of RAM... The changes I'm doing now er on the side of making the index as light as possible by default, and then users can turn on indexing per-module and get it set in a way that works for them.

The FG scripting system wasn't intended for this sort of use, so it's a bit of a balancing act to get it working well.

[Ludd_G]

No worries, I'd rather have people trying to push what FGU can do, than have it go stale. The competition from other VTTs is going to get intense going forwards.

Cheers,

Simon

[sirnoobsauce]

I pushed a new version which defaults to having no modules indexed, and basic indexing enabled. There is a new "Module Indexing" button in the options menu to open a basic interface where you can select which modules to index. It will also show the module indexing time and how many records were indexed for modules that have been indexed previously.

When you save&close on the window, it will re-index the modules. I figure with this, the out-of-box impact on loading campaigns should be minimized, and you can see from the "Module Indexing" window (once they have been indexed at least once) how long each individual module is taking to index, and how many records were indexed from the module.

[scoot138]

You are a rockstar!
Being able to select the modules I want indexed, it took I am guessing less than 10 seconds, when performing a search, it is almost instantaneous!
This is a game changer for real, I cannot wait to test it out in a session! You have just added a QoL that FGU has needed desperately!

[scoot138]

You need a donate button.

[spencerg]

Bravo - I'll poke around later - thank you so much for your work on this!

[damned]

You need a donate button.

He will need to sign up to sell on the Forge and then the Donate button will become available.

[sirnoobsauce]

FYI for anyone that has tried the extension and had issues with the indexing being slow, I am testing out a new potential version that uses a hack to implement an async event loop and move all the indexing to run asynchronously via the event loop. If you have had any issues with slow indexing, give this a try:

https://github.com/bakermd86/FoogleBrowser/releases/tag/v1.2.0
(https://github.com/bakermd86/FoogleBrowser/releases/tag/v1.2.0)

I have been trialing it with some of the users who have had issues with the indexing performance, and it seems to work a lot better. I am still testing it to make sure it is stable (the way I am implementing the event loop is a bit of a wonky hack so I want to make sure before I make the the live version on the forge). But so far, I haven't seen any major issues.

[Moon Wizard]

Just make sure that you don't use the GlobalRegistry/CampaignRegistry for any non-option data storage.

Regards,
JPG

[spencerg]

In regards to this - your previous version, while slow to loadup, was working perfect for me personally.

This new async-based version today (3hrs) ago, I'm stuck on the loading screen for 10 minutes.

I was testing out both async versions on GIT yesterday, and experienced various inconsistent behavior.

Sometimes it would load, get into the table and run the index fast. Sometimes it would get in, and run indexing painfully slow, sometimes i just had to kill the app.

In any event, I didn't mind the linear synchronous indexing/slow loadup - if it means stability on my end. Currently, its just stuck (5 mins now)

*Update* Now its in. Hmmm.

I'll keep hammering. Perhaps my 12 GB of RAM on this box isn't enough. You did merge the Async version into the HEAD right sirnoob ?

The one in which you're polling using a timer based implementation to simulate an asynchronous execution.

Once again though, I have to say, this is hands down the best workflow extension I've ever used for FG, for both its Modal feature (tabbed records under one modal), and its fast search.

[sirnoobsauce]

I pushed a new version today that includes an implementation of a background event loop to handle the indexing of database nodes so that the indexing doesn't slow down loading campaigns.

I also made all of the indexing behaviors configurable so that you should be able to have it working the way you want in any environment. I will update the first post with instructions on how to tweak the settings if desired, but the default behavior is an automatic mode that tries to adjust the event loop priority to index as fast as it can without impacting the responsiveness of the client itself.

Big thanks to spencerg for his help testing, as his environment was much more prone to performance issues than either of mine, so his feedback was very helpful for getting it working well across a wider range of conditions.

[spencerg]

Absolutely  awesome- my advice to everyone using this amazing workflow tool, play with the settings. I personally use blocking mode on load, with advanced indexing (basic off) and everything else on. Also, check your modules, do you really need all 50 of those loaded :)? The various settings and ability to prune what gets indexed and what doesn’t is a game changer as well. In summary, find the settings that work for you. It also should be noted, this extension is a lot more than just a tool to search. The ability to create custom containers of records is a huge improvement regarding organization and the ability to get at things you need quicker, without 30 different hotkeys. In summary i think it’s one of the best 3rd tools around. Really stellar job sirnoob! should be integrated. If you need any other testing i’ll be on the GIT repo!

[Morenu]

Starting to play around with this extension.

Does the indexing save locally? or every time I log in to the same campaign, will it start fresh and take the same basic amount of time?

[Morenu]

Also, could you give more detailed explanation for each of the setting options? (Some I get and some I am not sure on)

57119

[Morenu]

Your last update fixed a lot for us. I can start using FG normally after about 4-5 mins after clicking Join and Start. prior to yesterday's update it could take 20+ minutes (unsure as I left the computer after 15 minutes and returned 25 minutes later so 15-40 minutes). FYI, we use about 45 extensions and have 18 Modules loaded on the player side, probably 25 on the GM side plus 15 - 20 map packs

Now to play around with it and see what it can do :) thanks for the time you have put into it.

[sirnoobsauce]

Starting to play around with this extension.

Does the indexing save locally? or every time I log in to the same campaign, will it start fresh and take the same basic amount of time?

I don't persist the index, initially I had played with that, but storing the index has significant consequences that there isn't a good way to remedy, so I focused on just trying to mitigate the impact of indexing by implementing the background execution loop. The time taken can vary quite a bit based on a lot of factors, but generally speaking it will be roughly similar each time.


Also, could you give more detailed explanation for each of the setting options? (Some I get and some I am not sure on)

"Basic Indexing" does 2 things to speed up indexing speed. #1, it skips all database fields that aren't basic strings. #2 it does no secondary lexical tokenizing. So as an example: with basic indexing turned off, a search for the word "Indexing" would capture records containing the word "Indexing" but also the word "Index". While with basic indexing turned on, the search for "Indexing" would only match records containing the word "indexing" and not the word "index.

"Event Loop Priority" changes how many records the background event loop tries to process for each "tick" of the psuedo-event-loop. On lower settings, indexing will take longer but will have less impact on the responsiveness of the UI. On higher settings, indexing will be faster, but the UI will start to slow down and become unresponsive. On Automatic, it tries to index as fast as it can without negatively impacting UI responsiveness.

"Index Formatted Text" and "Index Non-Text Nodes" turn on/off indexing for the formattedtext (former) and dice, number, image, (latter) node types. When turned off, indexing is faster.

"Index on-load" controls whether or not the index is built automatically when the campaign is loaded. If turned off, you have to manually click the "Rebuild search index" button before the index is built.

"Show Indexing Status" controls whether or not the little window in the top right showing the status of the indexing jobs is shown.

[spencerg]

My new favorite settings are.

Basic Indexing (Because 9 out of 10 times its good enough) -

Index on load using 3.

I did notice something odd today using automatic, I got a runtime.

With all the problems everyone is having after yesterdays patch, it wouldn't surprise me at all it broke something.

I changed it back to 3 - ran same steps, no problem.

If it happens again I'll let you know, but it might be worth a sanity check or too.

[JeffDM]

I just want to make sure I understand - the browser can index everything within a campaign (spells, items, characters, character and party inventory, the pre-made adventure that is loaded, etc) and you can simply search in one place? Because that would be very worthwhile.

[sirnoobsauce]

I just want to make sure I understand - the browser can index everything within a campaign (spells, items, characters, character and party inventory, the pre-made adventure that is loaded, etc) and you can simply search in one place? Because that would be very worthwhile.

Yes, that's correct. Modules are not marked for indexing by default, just to avoid performance issues for people with many, many modules loaded. There is a "Module Indexing" button in the options menu to select which modules to index.

Searches in the browser will search all indexed campaign records, module records, and library reference pages.

[viviolay]

Hi!
Really enjoy the extension - thank you for making it. I did catch a bug (unsure if it's already imported). I was able to replicate it on a clean campaign with only this extension installed.
If you use the diagonal arrow icon in the upper right corner of an image window (just a regular one not associated with the FG browser or anything) to push it to the background of your FG, the button to bring it back out (opposite diagonal) or to push it to cover everything (same diagonal) stops working. So the image is permanently stuck.

[spencerg]

I concur with the post above. I noticed this bug the other day. The z-index gets screwed up or something similar when you send the map to maximize. Its like the invisible processing/indexing frame is overlayed on top and you cant click the controls at that point :)

[Tideking]

I also noticed an issue with expanding maps / images. I've attached an image with the broken menu circled in red.

[LeoKeros]

I also noticed an issue with expanding maps / images. I've attached an image with the broken menu circled in red.

My players just encountered this same issue.