4E Module Generator - Offline compendium > Fantasy Grounds

[VegaFontana]

Hello everyone,

This post is about a set of scripts that will allow you to generate Fantasy Grounds modules from the Offline Compendium data.


Foreword

https://i.imgur.com/dPtisxE.png

What is the Offline Compendium?
For those who never heard about the Offline Compendium, it was a way to access the Online 4E Compendium and with it, all official 4e content ever created without internet access.
It was great for us since we have a tradition of playing in a large farmhouse without wifi (+ it has way, way better research feature).
Nowadays since the online compendium has been shut down by WotC, it became the only way to access all 4E data and many DM made the migration before the shutdown.
Here is a video for more details https://www.youtube.com/watch?v=aNDze9Ok5fE).

But you know, there is several compendium existing, how do I know this is the right module generator?
Indeed, there is two compendium hosting 4e data these days. Or at least, two that I am aware of. These module-generating scripts will only work with the "Offline Compendium".
If your compendium looks like this picture below, you are in the right place. If it's not, you might want to check my other thread here https://www.fantasygrounds.com/forums/showthread.php?60524.

https://i.imgur.com/nGEYnOs.png

How does it work and which modules will I be able to create?
The scripts I bring here will allow you to generate modules sufficient to launch any campaign of the fourth edition in Fantasy Grounds.
We're talking about 5000+ NPCs for you DM and for your players of 9000+ Powers and 3000+ Feats ready to drag and drop in their character sheets.
Moreover, this can also generate modules including all non-magical base items, gear ready to take and armors/weapons available to be customized into magic items.


Installation & Usage

Ok so now how can I make it work ?
It should be easy, or I least I tried to make it easy.
First things first you need to download and unzip the module generation scripts here : https://drive.google.com/file/d/1ObtwSFT_eWmWvAzdHcTp8NuuK0mATvKq/

And follow carefully the detailed steps written below.

1. Copy the armor, feat, implement, item, monster, power and weapon folders from "PathYoYour4eCompendium/4e_database_files/".

2. Then paste them inside the sources folder of the generator located at "4E Module Generator/sources/".

https://i.imgur.com/UbktET5.png
(Or you could go the barbarian way here and copy all the folders from within the "4e_database_files" of your 4E Offline Compendium into the sources folder).

3. Execute the "4E Module Generator/_run_all.bat" file (double clic should be fine). Then six executable will be launched and six consoles should appear (once per modules generated) with info of the parsing happening.

4. Each consoles will ask you to press enter to close them once finished.

https://i.imgur.com/4XygupA.png

5. Finally you will be able to find the generated modules in the "4E Module Generator/export" folder in each specific repositories (i.e. "4e_NPC.mod" in "4E Module Generator/export/npc")

6. Just copy these 6 modules from their export folders into your FG modules folder (default should be ".../Fantasy Gounds/Data/modules") and load them in your campain!

7. Don't forget to give access to your players to the basic items / feats / powers modules to ease character creation.

https://i.imgur.com/xwpRear.png

And voilà you should be done and ready to start creating colorfull characters and dreadfull encounters!


Last notes

I'm curious, how does it works?
Remember when you copied the folders at step one? Each of these folders cointain `dataX.js` files with X beeing a number always ranging from 0 to 19.
These files are the source information used by the Offline Compendium and are in fact the only thing you need for this converter to work.
Each individuals scripts (*.py files compiled as *.exe files) are called by the `_run_all.bat` executable to parse text (json objects) from the Offline Compendium into Fantasy Grounds Compatible data which is then exported (in xml) and bundled in a module (mod).

Since it's only a handfull of python scripts (not compiled language), anyone can open these to see more how it works.
Or even change them into other generators for those of you with enough motivation (licence availaple in COPYING.txt).
Yet it's far from state of the art code since it was done quickly and also part of the first experimentation of a friend with coding. But hey, it works! (or should, or at least it worked here? Tell me if it does not!)

Any known issues?
- Sublass of items are swaped with Class and therfore automation for ranged weapons and armors does not work properly, it's an easy fix for a future update.
- Recharge for monsters skills are not tracked, this is due to the source data not storing this info at the same place every time. An upgrade of the npc script should put things back in the right order.


I sincerely hope this can help many of you the way it helped me setting up my last campain!

[VegaFontana]

Oh dear but I'm missing all my duty, I just realized that I didn't test the tool on another computer than mine!

I'll do that in the upcoming days to make ensure there is no problems with it. In the meantime I'll be around if any issues are encoutered :)

[YellowAdder]

Like others in the community, your work on this is appreciated!

I've tried it and ran into an issue. For each of the 6 modules, it gives this:

"python.exe System Error

The code execution cannot proceed because python37.dll was not found. Reinstalling the program may fix this problem."

I had a copy of python37.dll and put it in place to see what happened. The errors change to this:

"python.exe Application Error

The application was unable to start correctly (0xc000007b). Click 'OK' to close the application."

I tried a handful of other things including installing the latest VS C++ Redistributable (17/19) as well as the x64 and 32 versions of Python. Same issue. Not quite sure what it might be on my end for the moment. (I have tried this on two separate Win10 computers but they are largely identical. Same issue though.)

[VegaFontana]

Ah yeah that's what I feared.

I replaced the bat executing code using virtual environment with packages executables (.exe files).
It's heavier and less clean but it does work for now (at least I managed to generate modules on my sencond computer that had encountered this dll error).

Tell me if it's fixed on your side too!

( Link is updated on original post but you can get it here too: https://drive.google.com/file/d/1yAYO-KsJJJYDi-DPxFxeCzPTQ6f6PaLF/view )

[YellowAdder]

I'm no longer getting the previous errors, but the windows that arise to process each module disappear quickly and none of the data is processed as expected. Feels like it's on the edge of working though!

I don't think I'm doing anything wrong with the process, but I'll let you know when I can fiddle more and can try again on another machine or two. I could be user error! :)

Thanks again!

[YellowAdder]

Tell me if it's fixed on your side too!

I've now tried on another system with the same result: the .bat hits each of the .exes but they arise and shut down almost immediately. Maybe this will give you some hints, as I ran the scripts individually and saw these conclusions:

https://i.imgur.com/fS6RNSf.jpg

[VegaFontana]

Maybe this will give you some hints

Yes it does indeed! From what I see, it looks like there might be different versions of the Offline Compendium and different ways to store data :confused:

Let's make sure about that. Can you post the first and last parts of your data0.js of powers for instance?

Mine looks like this:

od.reader.jsonp_batch_data(20160803,"power",{"power11860":"<h1 class=encounterpower>Adamantine Blas .......... gon Magazine 377, page(s) 49.</p>"})

Additionally if you could tell me how "long" in terms of number of characters the file is?

[YellowAdder]

Can you post the first and last parts of your data0.js of powers for instance?

Here's mine:


od.reader.jsonp_batch_data(20160803,"power",{"power8000":"<h1 class=encounterpower><span class=level>Adept of Whispers ......... in Arcane Power, page(s) 115.</p>",})


Additionally if you could tell me how "long" in terms of number of characters the file is?

VSC reports 90399 characters.

EDIT:

Looks like it may be the appearance of spans that are different. Here's what I found for your entry:


od.reader.jsonp_batch_data(20160803,"power",{"power11860":"<h1 class=encounterpower><span class=level>Adamant Instructor Attack 11</span>Adamantine Blast ........

[VegaFontana]

Oook so this is about to get interesting, the data files syntax and size change indeed (even slightly) depending on the Offline Compendium (of course it does. -_-).

I will work on a fix for your version and make this as generic as I can so that it should avoid the isse from reproducing.
(Yet, I hope there is not too many changes on source html because I use these tags to generate the data for modules. If there is other changes, you might encounter other errors in the extraction process but let's not get negative too quick).

Good news, I might have a fix for you while I'll be reworking the scripts to be more generic.
It's a longshot and it will be a bit tedious. You need to open each one of yours dataX.js files and change the end of the line from ,}) into }) (remove comma will make it becom "valid json").
Maybe just start with the feats folder since it's one of the less complex script and if it works do it for all of them.

[YellowAdder]

Because of course it does. LOL.

I attempted to remove the extra comma for feats and it *did* process before running into an issue with the imports:

https://i.imgur.com/M7jMPbw.jpg

Still, it's moving in the right direction!

[VegaFontana]

Ok so about the first issue my hunch was accurate: in first versions of the Offline Compenium, a trailing comma remained in all dataX.js files.
Since this is not valid JSON syntax he must have had changed it in following versions since this can cause crash on some browsers.

Now about that 'bs4', the last generated versions of exe files should come with it bundled.
I even tested it on a friend's computer (Windows 10, without any development environment inside) just this morning and it worked well. So I don't get why it won't for you :(

Would you mind re-download the whole project zip and retry launching the exe file, first without and then with the dataX.js files in the source folder?

[YellowAdder]

Alright, trying now ... first having ago without any data files, I receive the expected "NO DATA FOUND IN SOURCES" error.

Next, moving the data (without end commas) in place ...

https://i.imgur.com/eAuUZO9.jpg

I only attempted Feats because I hadn't removed commas in the other files yet. It produced a 4e_Feats.mod! Next, I tested the mod in FGU and it provokes this error on loading:

https://i.imgur.com/4Y8p0dB.jpg

Suspecting something erroneous with the XML, I took the mod to another system and tested it with FGC. That works! I can see all the feats and they appear to work on characters.

The XML parser for FGU is stricter, as I understand it, and undoubtedly explains this issue. I also presume that the other modules will now build (maybe?) into modules. When I have time I'll give that a shot, but we might also suspect that it won't work for FGU even if it does for FGC.

Which begs the question, have you used yours on FGU? Is it just this older version of the compendium that has the invalid XML bumps for use in FGU?

[VegaFontana]

Which begs the question, have you used yours on FGU?

Yes of course, we are FGU users here. You can see on that screenshot the line of sight and modules loaded!

https://i.imgur.com/jOy033Y.png

So my guess would be that here too, the different source file yield to results with errors.
Could you send me your dataX.js files and generated feat module through private messages so that I can search what's wrong with the generated file (and what can cause it)?

This way I'll be able to update the scripts to work properly with older versions of the Offline Compendium :)

And by the way, thanks for your continued tenacity!

[YellowAdder]

'Pinging' you here in case you weren't getting private messages for some reason. If you're just busy, no worries, I just wanted to make sure you didn't think I fell off a cliff or something. I do have some updates on what I've tried, to mixed result. If you didn't get the PMs, let me know. Thanks!

[qunitvineu]

Does anyone know if there is a way to download the offline 4e compendium without having subscribed to the ddi back in the day? The place to pay for a new one they talked about in the video Vega linked doesn't work :(

[VegaFontana]

I just wanted to make sure you didn't think I fell off a cliff or something.

Giving you some update here, while I'm not done yet with the changes in the scripts, I have several good news. The scripts will now automatically clean trailing commas in dataX.js files to prevent crash on launch. I also updated feat script to be able to produce valid xml replacing '&' chars with '&amp;' (don't know why it worked on my fgu though). And the power script is now able to retrieve the right names for each powers.

I also narrowed why the npc scipt was failing with older compendium data. The source is widely different in some cases so I'll have to code different measures to integrate them properly. For now when it tries to integrate what I call these "legacy monster structure" it fails in recovering init/hp/senses/defenses/speed so they are useless. This will take a little more time but hey, at least it seems that the part of the script that recovers the monster's powers is working well for all (it would have been a huge pain if I had to update these parts too ^^').

[VegaFontana]

Does anyone know if there is a way to download the offline 4e compendium without having subscribed to the ddi back in the day?

Well sadly there is no longer any legal way to generate an "Offline Compendium" since DDI subscriptions are down.

But of course when there is a need and no legal way to fulfill it, then it calls for some gray areas. I know that versions of the "Offline Compendium" already generated are passing from one hand to the other between reddit users. I think I even saw someone hosting an online version of the Offline Compendium (how ironic, right?).

As you can see in my first post, only the database dataX.js files from the compendium are needed for the generator to work. I have a little idea on how ones could manage to download the whole data from the currently hosted edition or try to fetch precise links on how to acquire the compendium nowadays. But that would be considered fencing, wouldn't it? Therefore I can't really delve into that path and it's a quest that you'll have to undertake on your own.

[YellowAdder]

Fantastic news, Vega!

Having told one of my regular groups about your work on this, they agreed to plunge into a big 4E campaign. So, it's exciting to see the bumps coming out of the older format to pave the way for it. I'm sure others will find all this useful too!

[VegaFontana]

Having told one of my regular groups about your work on this, they agreed to plunge into a big 4E campaign. So, it's exciting to see the bumps coming out of the older format to pave the way for it. I'm sure others will find all this useful too!

And no later than today ! ;)

I have finished a version that parse the modules properly, based the source files of your version (as I said earlier, the trailing comma issue, the malformed xml, the statblock data table for npc, etc.).

You can download it now here (or from the main post I updated the link):
https://drive.google.com/file/d/1ObtwSFT_eWmWvAzdHcTp8NuuK0mATvKq/

I tested it on my computer and a friend's computer, I also parsed the generated db files: they are recognised as "valid xml" so this should deal with FGU issues too ! Thanks again for handing me everything I needed to upgrade the scripts to cover more versions of the Offline Compendium, hope you guys will have fun with 4e as much as we do here!

(And as always, I'm availabe to tacke eventual issues you might have on your side)

[VegaFontana]

Oh and while we're at it, could you please give me the version number of your Offline Compendium? I will add on the main post a list of versions against which the module generator has been tested and proved working.
This might help people knowing which version of their Offline Compendium have the most chances to work well (even though I think we tackled most of the possible issues here).

You can find this number at the end of the "Help" page:

https://i.imgur.com/jbEERub.png

https://i.imgur.com/IHHhZGB.png

Thanks in advance :)

[YellowAdder]

My version shows: 3.5.2.1 2016-12-19.

I'm going to grab your latest and greatest and will give it a spin when I get a chance. Can't wait to check it out. Thanks again for all your work on this, Vega!

[qunitvineu]

I have been able to find the "Portable Compendium" which I guess is a little different but provides the same service as the Offline Compendium. The database files for this one are in sql. Forgive my ignorance, I don't really know anything about programming, but I want to make sure. Data is sql form won't work with your scripts right? Also I'm guessing there probably isn't a way to turn these files into js ones either?

[VegaFontana]

My version shows: 3.5.2.1 2016-12-19.

Thanks for the version number!


I have been able to find the "Portable Compendium" which I guess is a little different but provides the same service as the Offline Compendium. The database files for this one are in sql. Forgive my ignorance, I don't really know anything about programming, but I want to make sure. Data is sql form won't work with your scripts right? Also I'm guessing there probably isn't a way to turn these files into js ones either?

Oh thanks for the heads up, I did not knew about this second "Portable Compendium" ! Which is ironically less portable than the "Offline" since it runs as an executable. Quite easier to find indeed.
SQL database are quite different when compared with the JS ones from the Offline Compendium for sure. But they are also know to be much more structured (this is what the S means in SQL, structured).

So it should be easier for me to create a whole bunch of new scripts that convert these "Portable Compendium" data files into Fantasy Grounds modules.
Mind that these ramblings are just some guesses I have now early in the morning and that the task might prove itself more tricky to do, but I have hopes this can be done easilly.

I guess it calls for more work then ! ;)
Once done, I'll make an other thread "Portable compendium > Fantasy Grounds"

[qunitvineu]

Thanks for the version number!



Oh thanks for the heads up, I did not knew about this second "Portable Compendium" ! Which is ironically less portable than the "Offline" since it runs as an executable. Quite easier to find indeed.
SQL database are quite different when compared with the JS ones from the Offline Compendium for sure. But they are also know to be much more structured (this is what the S means in SQL, structured).

So it should be easier for me to create a whole bunch of new scripts that convert these "Portable Compendium" data files into Fantasy Grounds modules.
Mind that these ramblings are just some guesses I have now early in the morning and that the task might prove itself more tricky to do, but I have hopes this can be done easilly.

I guess it calls for more work then ! ;)
Once done, I'll make an other thread "Portable compendium > Fantasy Grounds"

Vega, you are a saint

[VegaFontana]

Just trying my best, I'm glad if I can create things that helps people :)


It should be easier

Eh! Just oppened the sql database files, turns out only a few things are properly structured in it (names, level, categories).
But most of the core data is dumped as html code directly in a big text field (in a same manner but also different to the JS ones).
So it might be trickier than expected, on the up side I might be able to re-use some existing code, we'll see!

[YellowAdder]

I've used the new scripts and they produced all the modules as expected. That was a good to see happening. :)

The results inside FG do lead to questions, some of which may be a result of the older data I'm going to guess.

For example, leather armor has Class: Leather and an empty field for Subclass:. To work properly, it should have Class: Armor and Subclass: Leather Armor. Without those, the item never registers as armor and no armor bonus or skill check penalty is applied.

Weapons have a similar scenario. A battleaxe from the parse lists "Class: Military one-handed melee weapon" and "-Axe" for the Subclass. I think it's supposed to have "Class: Weapon" and a Subclass of "Military one-handed melee weapon".

An example I tested where this shows a difference is by making an attack with a sling against a prone target. With the default parsed weapon, it treats the attack like any other and applies the defense penalties as if it were a melee attack. Updating it to a version with the proper subclass gives the prone target the improved defense vs. ranged attacks.

The parsed properties data may be an issue as well if it's supposed to be an abbreviated comma-separated list as with 5E. (e.g., versatile, silver, magic) I don't know how it's supposed to look for 4E
without some testing though. There is a page for 5E item values (https://www.fantasygrounds.com/wiki/index.php/5E_Item_Sheet) but I can't find a similar one for 4E, unfortunately.

Of course, this could be a result of the older compendium data being different from your later version? Are you seeing these same outputs from yours?

[VegaFontana]

Could be a result of the older compendium data being different from your later version? Are you seeing these same outputs from yours?

Both could be questionned here, but I think the issue might come from the scripts here. Not sure if it was clear enough but the items scripts were an opportunity for me to introduce a friend to software programming.
Therefore I have not coded these and neither did a thorough test of his production since the other players did not see any issues (Things like high crit working properly, etc.).

But the armor and prone examples you bring up are interesting and I'll have him revise his scripts, test them properly and release a new version.
In the mean time you might have to edit the entries you want to use.

To make sure the update is also fully compatible with 3.5 as well as 3.6 Offline compendiums.
I would be glad if you could email me your dataX.js files for items so he can work on both sources :)

Nevertheless I'm glad you managed to get the modules working!

[VegaFontana]

I have been able to find the "Portable Compendium"

It was no picnic, but then again, when things start to work out well it's a real relief and plenty of satisfaction.
Consider yourself served my friend! On the following thread you will find an alternative to this generator, adapted to the "Portable Compendium".

https://www.fantasygrounds.com/forums/showthread.php?60524

Tell me if you have any issues using it!

[Gurlock]

Wow, this sounds amazing!

I've decided to go back to 4e after years with 5e, and am faced with the arduous task of getting FG working with it again. How do I obtain the offline compendium in the first place now that DDI is dead? The install page talks of logging in with your DDI credentials so it can pull data.

[VegaFontana]

How do I obtain the offline compendium in the first place now that DDI is dead? The install page talks of logging in with your DDI credentials so it can pull data.

Sadly you can't get a new subscription since they shut down the DDI service. But you can sill find links to compendiums online (around some reddit / discord etc.).
But I can't really give you any links or any further help in finding those things / copy their source data since it would be considered illegal (I guess?).

Good lunk finding those, I believe you already have access to the "Portable Compendium" from your post on the other thread, so you are already in a good way :)

[TheUnforgiving]

I've got a database folder from the latest 3.6.1 version of the Offline Compendium and ran it through the Python scripts. I didn't encounter any obvious errors, however the console output is giving me some concern. For NPCs, Feats, and Powers, all entries that were recovered from the database were converted, but other numbers don't match up:
* 631 basic weapon entries were recovered, but only 128 were converted
* 493 basic armor entries were recovered, but only 99 were converted (as Weapons)
* 1963 feat entries were recovered, but only 180 were converted.

Is this intended behavior?

[Eyeofthenyte]

Hello, I've been trying to convert my offline compendium js files. I keep getting an error in each of them when running them individually.
47956
if needs be I can provide the data0.js as an example.
I double checked and have some text formatting in the js files. Mostly carriage returns and tabs. I tried taking them out to now effect.
The error itself seems to be a problem in the python file. But I'm not sure what the displayed error is telling me.

[qumaciel]

I've managed to generate the .mod files, but inspecting the db.xml I see there is no <library> tag, so how can we access the power list, for instance?

Edit: Solved with Better Menus extension.
https://www.fantasygrounds.com/forums/showthread.php?58226-Better-Menus-(CoreRPG-5E-etc)
You can see Powers/Feats in the Records menu.

[Krabbe]

Is it just me, or does the Adventure Vault I & II along with Mordekainen not show up in basic_items, basic_armor, and basic_weapons? If this is intended, is there a reason for it to be so? I would love to have every item accessable for my games.

Thanks for the great work on the module generator! It has enabled me to play 4e with 4 of my friends, and it runs quite smoothly.

[beastman013]

I've run the program. Got the files to convert/create the .mod files. The file sizes are small, and I cannot see the detailed info. Is there a chance someone can supply the .mod files for group use?

[Jerry Taylor]

Ran the program and everything came out swimmingly. While I'm thrilled beyond words (I came to 4e after the DDI shut-down), it would be wonderful to also have the program cover race, class, rituals, and combat stuff like Opportunity Attacks, Basic Melee, etc. Any chance that might be covered someday?

[thedrewboat]

55766

Any reason this is popping up? Or a known fix?

[garymichaelduke]

Hi,
i was refered here because i am attempting to get the full functality out of FGU 4e. i need the menu tabs for race and class. all i need are the structues to be filled in and used. i dabbled in programming some 25+ years ago. the details are a bit fuzzy at this distance. i am only intersted in using the 4e ruleset for personal use in a solitare. i have be advised that i need extentions for these items. i ama total nube when it comes to creating these.
a little background on me: i have been a DM for the last 40+ years. when 4e cam out i picked up a PH, saw that 1st level characters were geven powers. closed the book and wen back to running the 3e campaign that we were playing. now in my dottage i have decided to explore 4e w/ FGU only to find it is crippled. any not too technical help you can give will be most appreciated.

[Griogre]

The 4E ruleset is very functional.

garymichaelduke back in the days before the the defunct rules compendium came out we input the data by hand. I'm telling you this because for first level characters you only need a couple of at wills, an encounter and a daily power. So like 4-5 powers per first level character and maybe a racial.

If you are just doing some solitaire stuff that is still probably the best way. By the time 4E was finished there were thousands of powers. It makes very little sense to input anything like that amount since you will only use a handful even with full 30th level characters.

Where you will run into problems is monster data because there are thousands of them too and unlike powers you are going to need a fair amount of them. Monsters were designed to act much like parties with various monster types complimenting each other and increasing and decreasing difficulty due to synergies. Monsters were generally simpler since there was such a variety of them for each monster.

My best advice to you is to just get some second hand books cheap off of e-bay or out of a used book bin and play your game on FG with the books open in your lap and skip the data entry.

However, if you do want to do data entry if you enter a few powers into a character sheet and monsters in as NPCs and then you could look at how the XML data is structured.

[Starwolf]

I have tried to download this converter from the link provided in the first post, linked to google drive. It seems like Google have this link blocked due to some legal issues. Is there a place where I can get this program downloaded?

[SieferSeesSomething]

I have tried to download this converter from the link provided in the first post, linked to google drive. It seems like Google have this link blocked due to some legal issues. Is there a place where I can get this program downloaded?

Valkyrion set up a github repository (http://fantasygrounds.com/forums/showthread.php?60524-4E-Module-Generator-Portable-compendium-gt-Fantasy-Grounds/page6) but I think you need the Portable Compendium for theirs. But if you have it, it works great, at least last time I checked. It's been awhile. But it made 4e work amazing in Fantasy Grounds.

[Starwolf]

Valkyrion set up a github repository (http://fantasygrounds.com/forums/showthread.php?60524-4E-Module-Generator-Portable-compendium-gt-Fantasy-Grounds/page6) but I think you need the Portable Compendium for theirs. But if you have it, it works great, at least last time I checked. It's been awhile. But it made 4e work amazing in Fantasy Grounds.

Yes, this seems to work with a different version of the Portable Compendium which uses sql datafiles. The compendium version I use use .js files.

[CrackSensei]

Does anyone still have a link to these files to generate modules?