4E Item Parser

[valeros]

Parser updated to help create smaller modules. The only input accepted is the original text files left behind by the original parser. (Fantasy Grounds ".mod" files will not work.)

Requires .NET Core 3.1 (x86, not x64)

Output can be a single module or multiple modules, created all at once, broken down by the entry source (i.e., "Published in") or by hard-coded pre-defined categories. The module(s) must be manually moved to the Fantasy Grounds "module" folder.

All output is filtered by the Sources and Entry Types selected. Also, for some options of multiple modules, only certain entry types will be included. (For example, the option to create multiple modules for Powers by Class will only include Powers in the output modules even if other entry types are selected.)

When creating multiple modules "By Source", the Dungeon magazines are combined into one module by default. The same for Dragon magazines. There is an option to not combine magazines and have each magazine as its own module.

When creating multiple modules, custom module thumbnails can be used for each module. Simply replace the appropriate thumbnail(s) under the appropriate sub-directory of the "thumbnails" directory. (For example, for a custom thumbnail for the Monster Vault book, replace the "Monster Vault.png" file under the "thumbnails/by source" directory. Or for the thumbnail for Wizard powers, replace the "Wizard.png" file under the "thumbnails/powers by class" directory.) The name must be character for character the same as the existing file. (There is an option to override this behavior and use the same thumbnail for all modules created.)

When creating multiple modules, the names of the modules will be named by combining the Prefix and Suffix entered in the tool to the hard-coded name. The hard-coded name will be the same as the thumbnail name. For example, if you put "4E" for the prefix and "Powers" for the suffix and select Powers by Class, you will get modules named "4E Wizard Powers", "4E Rogue Powers", etc. (A space character is automatically added after the prefix and at the start of the suffix otherwise the above example would result in modules named "4EWizardPowers", "4ERoguePowers", etc.)

[valeros]

Requires .NET.

Also, remember that magical armor, implements, and weapons Compendium entries do not include the base mundane armor/weapon info so you will have to modify the magical item to match the specific mundane armor/weapon to which you want it paired.

Changed executable to leave items in a folder with the module name as well as attempting to create the module file. If the .mod file is bad, use the FGPEMUtility.exe utility from https://www.fantasygrounds.com/downloads/ to package the folder as a .mod file. (Be sure to select .MOD in the FGPEMUtility application because .pak is the default.)

[Torgaard]

eeeeee! :D

Can't wait to get home and try this. Been waitin' to parse Mordekainen's forever.

[primarch]

Hi!

DUDE! You gonna be one popular hombre for having done this!

Hats of to you!

Primarch

[primarch]

Hi!

Valeros, I have run into a interesting problem. I can successfully download and parse the selected entries, but instead of one "module" I get 4 files. A common and definition xml file a thumbnail png file and a mod file that is only 1 kb in size. The size of the common db file is almost 19megs, all the items on the compendium.

If I transfer just the 1kb mod file, fantasy grounds doesn't even recognize it, thus it does not appear on the module activation section. If I transfer all files to the proper modules folder, still no usable module to activation in the module activation section.

From expereince I know that a standard fantasy ground mod file can be extracted into the components above using some unzip program. However I cannot just "re-zip" them with your parser products into a workable module.

Is it supposed to produce just on file (not 4) or is there another step?

I would of sworn I have seen a xml to mod converter, but perhaps I am mistaken

Any help appreciated.

Primarch

[Trenloe]

However I cannot just "re-zip" them with your parser products into a workable module.
Yes you can - zip up the common.xml, definition.xml and thumbnail.png files into one file. Make sure you use ZIP compression not RAR or anything else. Also ensure you zip the 3 files, not the directory they are in. Then rename the .ZIP extension to .MOD.

Of course, it would be nice if the parser did this for you! :-) I'm guessing that there is an issue with this step at the moment as valeros mentions problems renaming from .zip to .mod in the OP.

[Dakadin]

Hi primarch,

You might be able to get it working by doing this:
* Rename the MOD file to have a .ZIP extension.
* Windows should open the ZIP file like a folder if you double click on it.
* Copy the other 3 files (common.xml, definition.xml, and thumbnail.png) to the ZIP file folder.
* Go up one directory level so the ZIP looks like a file in Windows Explorer.
* Rename the ZIP file to have a .MOD extension.
* Copy the MOD file to the modules directory.
* If FG is running, exit FG.
* Start FG and test it out.

[primarch]

Yes you can - zip up the common.xml, definition.xml and thumbnail.png files into one file. Make sure you use ZIP compression not RAR or anything else. Also ensure you zip the 3 files, not the directory they are in. Then rename the .ZIP extension to .MOD.

Of course, it would be nice if the parser did this for you! :-) I'm guessing that there is an issue with this step at the moment as valeros mentions problems renaming from .zip to .mod in the OP.

Hi!

Actually I did just that, but I erred in just adding .mod to the rename of the file, however I was just adding to the name not changing the file extension. Once I made file extensions under windows I changed it to .mod and it worked perfectly.

It's too bad you can't decompile the files made by this parser with DrZeuss decompiler (I tried, it seemed not to work) since I would use the decompiled files to substitute my original item files with these new ones an produce an updated mod file of whole books with these item files.

In any event this is a great tool and much needed. I downloaded all the items in 20 windows in under 5 minutes and the mod file actually nice broken down in convenient categories with makes one all-item book quite easy to navigate.

My humblest thanks to you sir for this magnificent tool! :)

Primarch

[valeros]

Changed executable to leave items in a folder with the module name as well as attempting to create the module file. If the .mod file is bad, use the FGPEMUtility.exe utility from https://www.fantasygrounds.com/downloads/ to package the folder as a .mod file. (Be sure to select .MOD in the FGPEMUtility application because .pak is the default.)

[Zeus]

Nice work Valeros. I will have a play with this later this weekend.

For those wanting to decompile the .mod this tool creates; I can make available the source for the decompiler if anyone wants to take on the development. I am a little maxed out with RL projects at the moment and if I am being honest am starting to wind down on 4E games (not for entirely but certainly with less frequency).

Same goes for any of the other tools I have made available over recent years; ht me up with a PM if anyone is interested.

DrZ.

[valeros]

Updates to item parser:

Updated to create Mundane Armor and Mundane Weapons as equipment so they can be combined with a magic armor, weapon, or implement to form a fully populated magic item.

Updated to also leave a file that can be parsed again without downloading if parsing changes and the option to parse these files from the GUI.

Removed automatic creation of .mod file due to issues. Use the FGPEMUtility.exe to form the .mod file from directory containing output files.

Fixed a couple of bugs.

[Griogre]

I got a chance to use this tonight and it worked fine on the couple of books I tried. I noticed a few things I thought were a bit strange and do have some feedback.

1) There are specialized mundane armor, weapon, and equipment tables that show cost and weight. I was surprised you didn't use them. Likewise for the Magic item tables.

https://i105.photobucket.com/albums/m222/Griogre/MundaneEquip.png

https://i105.photobucket.com/albums/m222/Griogre/MagicItems.png

2) In Mordenkainen's Magnificent Emporium, at least, the equipment is coming in as magic items instead of mundane items like the non magical armor and weapons.

3) It would be nice if you could specify the categoryname for the module so it could be listed with other similar modules like Core or Player's Supplement.

4) Likewise, it would be nice if you could specify an image to use as the module's thumbnail.

5) I would suggest you combine all magic armor, magic weapons and magic implements into one listing each instead of 3 each. If you don't know the exact level of an item having to look in 3 places instead of one means you seem to always pick the wrong one.

6) I'm not sure why you are outputting a common module. I'd personally be nervous about allowing people to distribute the WotC content. A client module might be better.

7) Personally, for MME I had the non magic stuff sort to the top and the Magic sort together but YMWV.

https://i105.photobucket.com/albums/m222/Griogre/LibPaneSort.png

[valeros]

Griogre, thank you for constructive feedback.

1) I do not know FG well enough to undestand how the underlying properties of the controls/files work with the specialized tables. If someone sends me an example file using the specialized tables, I might try changing it. (I did try playing with them, but not successfully.)

2) Fixed

3) Fixed

4) Fixed

5) Fixed with compromise. One listing if less than 1200 items would be in the listing (like Mordenkainen's Magnificient Emporium), otherwise three listings (like when downloading all items at once.) Since I downloaded all the items into one module, FG was not happy about the number of items in the table when I left armor, implements, or weapons as one listing each. The tables were not very responsive and scrolling was a pain. Breaking them into three worked well.

6) Fixed

7) Fixed

[Griogre]

You've been busy.

1) I'll put together something by hand (or more truthfully copying and pasting) but it might be next week.

5) FG really doesn't seem to deal well with a dynamic list of more than a few hundred. I think some people ran into the same issue when they decided to make huge books of powers or monsters from all the books. Its probably part of that extensibility vs. only native code trade off, but I'm pretty sure the original code base never conceived of library modules as large as some are now.

[Asturia]

How do we combine these item files with existing mods?

[Griogre]

Probably a lot of people will want to know that so I made a quick video on how. The basic idea is you take the data and library links from the item parser and put them in the XML for a module made by Tenian's Parser.

https://youtu.be/eesd7csoXjc

[valeros]

Added ability to parse Backgrounds, Deities, Diseases, Poisons, Rituals, and Terrain.

Also, you can combine these categories together into one module. You can add multiple entry types or select multiple files before downloading/parsing.

Items now use the specialized tables.

If you combine more than one data file (i.e., not the module data, the file left behind with the raw data), it creates a new combined data file.

The XML output now puts all data & indexes under one tag for a particular category (e.g., Backgrounds or Items.) There is a tag named the same in the library section. So copying the data into another module means copying the one tag with all the data & indexes and the same tag for the library. See Griogre's video for instructions on how to do that.

Diseases only work as good as FG allows. FG only allows three stages (not five), does not provide ability to handle variable level diseases (i.e., must list a value for the DC, cannot say "Easy DC".) Also, no space to list special info (like Abyssal Plague.)

[Griogre]

I got a chance to try out the new version and it works well. The only strange thing I noticed is that the sort by level for magic other than Magic Armor or Weapons is out of order because of level numbers less than 10, which you are basing your element names on, don't have leading zeros and ANSI sort.

As an example in PH Hands you have:

16 Shadowfell Gloves
26 Shadowfell Gloves
6 Shadowfell Gloves

IE 6 > 2 so it comes after 26. I'd suggest you insert a zero in all the element names with levels less than 10 so something like <ShadowfellGlovesLevel6> becomes <ShadowfellGlovesLevel06>

Edited: For clarity. Apparently, punch drunk from lack of sleep, last night... :p

[valeros]

Fixed item level sorting issue

[Griogre]

Sorry, I forgot it until I saw you reply. I've been meaning to mention the parser does odd things to an existing directory. It currently deletes them and throws the client in the next level up when running from a file. I think it just deletes everything in the directory if you select entries from Compendium. I'd suggest you make a check to see if the target directory already exists and if it does don't make a new one and just overwrite any the parser's output files. I would typically keep module icons and sometimes modules in that directory.

[tdewitt274]

I tried the Parser and seem to run into a couple of errors. I'm guessing they're the same as the status window indicates the same error. If it matters, I'm running Windows 8 Pro.

The parser seems to bomb on the Items scrape. It looks like all the entries may have been scraped (there is a text file), but the file is not parsed in any way.
I get the following error:
3679 Total Entries for Download & Parse
3679 Item Entries

Download Start Time = 1/5/2013 12:52:10 PM
Download End Time = 1/5/2013 12:58:29 PM
Download Total Time = 00:06:19
Vecna's Boon of Diabolical Choice: Not Parsed: <SPAN id=headerlevel class=milevel style="TOP: -24px">Level 0 Uncommon</SPAN></H1>


See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.NullReferenceException: Object reference not set to an instance of an object.
at ItemParsers.MagicItemParser.PopulateItem(MagicItem magicItem, ItemLevelInfo itemLevelInfo)
at ItemParsers.MagicItemParser.ParseItem(CompendiumEn try itemCompendiumEntry)
at ItemParsers.ItemParser.Parse(CompendiumEntry itemCompendiumEntry)
at _4eParser.AllEntriesParser.Parse(CompendiumEntry entry)
at _4eParser.AllEntriesParser.Parse(List`1 entries)
at MainViewModel.MainViewCompendiumModel.ProcessEntri es(String moduleName, String moduleCategory)
at MainViewModel.MainViewCompendiumModel.ProcessCompe ndiumEntries(String email, String password, Int32 numBrowsers, String moduleName, String moduleCategory)
at _4eParser.MainViewController.ProcessEntries()
at _4eParser.MainViewController.ProcessDownloadAndPar seEntries()
at _4eParser.MainViewController.DownloadButtonClicked (Object sender, EventArgs e)
at _4eParser.LogForm.DownloadButtonClicked(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18010 built by: FX45RTMGDR
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
4eParser
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Downloaded%20Files/Programs/Fantasy%20Grounds/Valeros4eParser/4eParser.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.17929 built by: FX45RTMREL
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.17929 built by: FX45RTMREL
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18016 built by: FX45RTMGDR
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.17929 built by: FX45RTMREL
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.mshtml
Assembly Version: 7.0.3300.0
Win32 Version: 7.0.3300.0
CodeBase: file:///C:/WINDOWS/assembly/GAC/Microsoft.mshtml/7.0.3300.0__b03f5f7f11d50a3a/Microsoft.mshtml.dll
----------------------------------------
Microsoft.CSharp
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.17929
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/Microsoft.CSharp/v4.0_4.0.0.0__b03f5f7f11d50a3a/Microsoft.CSharp.dll
----------------------------------------
System.Dynamic
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.17929
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Dynamic/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Dynamic.dll
----------------------------------------
Anonymously Hosted DynamicMethods Assembly
Assembly Version: 0.0.0.0
Win32 Version: 4.0.30319.18010 built by: FX45RTMGDR
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_32/mscorlib/v4.0_4.0.0.0__b77a5c561934e089/mscorlib.dll
----------------------------------------
System.Dynamic.DynamicAssembly
Assembly Version: 0.0.0.0
Win32 Version: 4.0.30319.17929
CodeBase: file:///C:/WINDOWS/Microsoft.Net/assembly/GAC_MSIL/System.Dynamic/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Dynamic.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.




When attempting to parse a specific file, I get an error:
Working, please wait
Vecna's Boon of Diabolical Choice: Not Parsed: <SPAN id=headerlevel class=milevel style="TOP: -24px">Level 0 Uncommon</SPAN></H1>
Working, please wait
Vecna's Boon of Diabolical Choice: Not Parsed: <SPAN id=headerlevel class=milevel style="TOP: -24px">Level 0 Uncommon</SPAN></H1>

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.NullReferenceException: Object reference not set to an instance of an object.
at ItemParsers.MagicItemParser.PopulateItem(MagicItem magicItem, ItemLevelInfo itemLevelInfo)
at ItemParsers.MagicItemParser.ParseItem(CompendiumEn try itemCompendiumEntry)
at ItemParsers.ItemParser.Parse(CompendiumEntry itemCompendiumEntry)
at _4eParser.AllEntriesParser.Parse(CompendiumEntry entry)
at _4eParser.AllEntriesParser.Parse(List`1 entries)
at MainViewModel.MainViewCompendiumModel.ProcessEntri es(String moduleName, String moduleCategory)
at MainViewModel.MainViewCompendiumModel.ProcessCompe ndiumFiles(List`1 fileNames, String moduleName, String moduleCategory)
at _4eParser.MainViewController.ParseFile()
at _4eParser.MainViewController.ProcessParseFile()
at _4eParser.MainViewController.DownloadButtonClicked (Object sender, EventArgs e)
at _4eParser.LogForm.DownloadButtonClicked(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

[tdewitt274]

Backgrounds also produces an error and crashes. No error message, just a force stop.

[gamerhawaii]

I see the issue. For this case, the parser was very picky and expects:
<SPAN style="TOP: -24px" id=headerlevel class=milevel>
not
<SPAN id=headerlevel class=milevel style="TOP: -24px">

[valeros]

Try this one for fixing the item part (not the backgrounds). It may work or at least should crash at a different point. Thanks

[tdewitt274]

I see the issue. For this case, the parser was very picky and expects:
<SPAN style="TOP: -24px" id=headerlevel class=milevel>
not
<SPAN id=headerlevel class=milevel style="TOP: -24px">
I was messing around with parsing data from the Compendium some time ago and found there are a lot of things like that.

I have some ideas that could be helpful when dealing with those types of issues. PM me if you're interested. However, it is in VBScript so some translation to the language being used would be needed.

[tdewitt274]

That seems to work. Looks like there is another one as well.

<SPAN class=miright style="TOP: -18px"><B>Time: </B>30 minutes</SPAN>


Thanks!

Todd

[valeros]

Please let me know if this one works. I also looked for any other occurences of the same type of parsing dependency for "SPAN" tags and did not see one.

[Biscuts]

Hey Valeros, Great work! It has worked flawlessly for me. However I am wondering if you are going to make it possible to parse Creature data. I know that Tenian's parser handles them, I just really prefer the way your parser sorts the data, and would prefer each compendium section as a single mod file. Thanks for your work either way.

[valeros]

Actually, the version of the parser that I have for myself parses the "new style" monsters. But I did not make that available because it does not parse the "old style" monsters. (I think it is 2135 new to 3035 old as of today.)

I call "new style" the new formatting (with sections for Traits, Standard, etc.) but also that the monster usually has the "new" stats. That is the adjustment made to monsters around July 2010 (and Monster Manual 3) that bumps up damage, adjusts defenses, etc.

I am not very excited to try to parse the old monsters. I think it would be a lot of work for not much return, especially since the monsters would then have to be manually updated to the "new" stats. (Although if I knew a good way to upgrade them to the new stats atuomatically when parsing it might be a nice feature.)

Also, the item parser has only been downloaded around 50 times, so there does not seem to be that much demand for it.

[Zeus]

Actually, the version of the parser that I have for myself parses the "new style" monsters. But I did not make that available because it does not parse the "old style" monsters. (I think it is 2135 new to 3035 old as of today.)

I call "new style" the new formatting (with sections for Traits, Standard, etc.) but also that the monster usually has the "new" stats. That is the adjustment made to monsters around July 2010 (and Monster Manual 3) that bumps up damage, adjusts defenses, etc.

I am not very excited to try to parse the old monsters. I think it would be a lot of work for not much return, especially since the monsters would then have to be manually updated to the "new" stats. (Although if I knew a good way to upgrade them to the new stats atuomatically when parsing it might be a nice feature.)

Also, the item parser has only been downloaded around 50 times, so there does not seem to be that much demand for it.

If your playing 4E I built in a tag (mm3mode) for NPC powers which instructs the ruleset how to parse the Powers of a NPC's stat block.

If a power has the mm3mode tag and its set to 1; the ruleset will parse as per MM3 monster formats( i.e. Traits, Hit:/Attack:/Effect: lines etc. etc.), if no tag is present the ruleset will treat the power stat block as per MM1/MM2 format.

[primarch]

If your playing 4E I built in a tag (mm3mode) for NPC powers which instructs the ruleset how to parse the Powers of a NPC's stat block.

If a power has the mm3mode tag and its set to 1; the ruleset will parse as per MM3 monster formats( i.e. Traits, Hit:/Attack:/Effect: lines etc. etc.), if no tag is present the ruleset will treat the power stat block as per MM1/MM2 format.

Hi!

DrZeuss, what utility contains this tag you built? Was it an addition to the Tenian parser?

On a side question the new 5e parser will it parse 4e data (scrapping as well) or will someone have to step up and add that?

Thanks!

Primarch

[Trenloe]

Also, the item parser has only been downloaded around 50 times, so there does not seem to be that much demand for it.
50 times! I think that's pretty good. :)

[Zeus]

Hi!

DrZeuss, what utility contains this tag you built? Was it an addition to the Tenian parser?

On a side question the new 5e parser will it parse 4e data (scrapping as well) or will someone have to step up and add that?

Thanks!

Primarch

Its not a utility, it was an enhancement I delivered originally via the 4E Enhanced Personalities extension. Moon_wizard then incorporated the extension into the base 4E and 3.5E FGII 2.8.x rulesets along with Enhanced Items, Partysheet, Calendar and Tables etc. etc.

As for PAR5E, yes it will be able to support 4E parsing (in fact it will support ANY rpg system) but first someone will have to create the appropriate library for the target FGII ruleset and routines for handling the rpg system specific data content and format. PAR5E will initially release with a 5E library and an example template library. As for scraping, if Wizards release an online Compendium for 5E, I will look to add support to PAR5E at a future date.

Once PAR5E and 5E are finished and released and if time is available (and there's strong demand for it) I'll see if I can release a library for 4E.

However, I have to ask, why would you want 4E to be supported by PAR5E? The existing 4EParser and Valeros's tool both seem to cope well enough with 4E content. The issues relating to embedded HTML tags is an impossible situation for the developers of these tools to permanently fix. The issue lies with Wizards and their haphazard approach to entering data into the Compendium with consistency. For a company that relies on the quality of the information published through the various mediums they employ, they do a very poor job of managing the content. I mean who stores data in HTML, seriously? Whoever made that decision at Wizards should be shot.

If I were in charge, I would have insisted that the data be stored as XML and then employ XSL/CSS to reformat to HTML for presentation.

[Zeus]

50 times! I think that's pretty good. :)

Also, remember its typically only DMs that will use the Parsers, so 50 x 4 average players per group, suddenly thats 200 users benefiting ...

Keep up the effort Valeros. In fact, would you be interested in developing the 4E library for PAR5E? I can make the PAR5E source script and modules (Perl) as well as the current WiP 5E library I am developing available as a reference. I would do it my myself but I have a list of projects as long as my arm not to mention the new 5E ruleset, so it would have to wait a bit before I could focus on it.

[Zeus]

Valeros - Another thought.

The 4EParser by Tenian appears to be no longer under active development. However functionally it still works well with current 4E data (aside from Item scraping which is slightly broken).

Instead of developing and supporting a tool that scrapes the items and then produces FGII XML, have you considered outputting the data to flat file text format.

If you use the format used by the 4EParser, the 4EParser will take care of producing the XML. The parsing functions still work, its just the scraping functions that have been impacted by DDI changes.

Benefits of approach.

- you don't have to worry about maintaining a tool to output FGII 4E specific XML
- you will be able to leverage the Item Tables (for example) with figuring out the XML structure
- users can easily produce 4E modules with all content inclusive

Just a thought. If on the other hand you want to add Table support to your parser, here's a breakdown of the XML structure you require.

- 4E uses the following nodes under the modules root XML to hold data:


<reference4EAA> - where 4EAA is the moduleid for the module
<weapondata> - listnode containing nodes of all weapon DATA i.e stats
<weapon1> - where weapon1 is the name of a weapon
...
</weapondata>
<armordata> - listnode containing nodes of all armor DATA i.e stats
<armor1> - where armor1 is the name of an armor
...
</armordata>
<equipmentdata> - listnode containing nodes of all equipment DATA i.e stats
<equipment1> - where equipment is the name of a weapon
...
</equipmentdata>
</reference4EAA>

The second root node holds the data for the window list (or Table):

<equipmentlists4eAV>
<core4eAV>
<description type="string">Equipment Table</description>
<groups>
<section001>
<description type="string">Item Category/description>
<subdescription type="string">Item SubCategory</subdescription>
<equipments>
<equipment1>
<link type="windowreference">
<class>referenceequipment</class>
<recordname>reference4eAV.equipment.equipment1@4e Adventurers Vault</recordname>
</link>
<name type="string">Equipment Item 1</name>
<weight type="number">0</weight>
<cost type="number">0</cost>
</equipment1>
</section001>
<section002>
...

[primarch]

50 times! I think that's pretty good. :)

Hi!

I agree. A community that represents a niche within a niche (as in people whom play RPG's AND play them online), 50 is pretty good.

I for one have greatly enjoyed Valeros contribution and would hope he will be enthusiastic in creating more things such as his item parser.

Primarch

[primarch]

Valeros - Another thought.

The 4EParser by Tenian appears to be no longer under active development. However functionally it still works well with current 4E data (aside from Item scraping which is slightly broken).

Instead of developing and supporting a tool that scrapes the items and then produces FGII XML, have you considered outputting the data to flat file text format.

If you use the format used by the 4EParser, the 4EParser will take care of producing the XML. The parsing functions still work, its just the scraping functions that have been impacted by DDI changes.

Benefits of approach.

- you don't have to worry about maintaining a tool to output FGII 4E specific XML
- you will be able to leverage the Item Tables (for example) with figuring out the XML structure
- users can easily produce 4E modules with all content inclusive

Just a thought. If on the other hand you want to add Table support to your parser, here's a breakdown of the XML structure you require.

- 4E uses the following nodes under the modules root XML to hold data:


<reference4EAA> - where 4EAA is the moduleid for the module
<weapondata> - listnode containing nodes of all weapon DATA i.e stats
<weapon1> - where weapon1 is the name of a weapon
...
</weapondata>
<armordata> - listnode containing nodes of all armor DATA i.e stats
<armor1> - where armor1 is the name of an armor
...
</armordata>
<equipmentdata> - listnode containing nodes of all equipment DATA i.e stats
<equipment1> - where equipment is the name of a weapon
...
</equipmentdata>
</reference4EAA>

The second root node holds the data for the window list (or Table):

<equipmentlists4eAV>
<core4eAV>
<description type="string">Equipment Table</description>
<groups>
<section001>
<description type="string">Item Category/description>
<subdescription type="string">Item SubCategory</subdescription>
<equipments>
<equipment1>
<link type="windowreference">
<class>referenceequipment</class>
<recordname>reference4eAV.equipment.equipment1@4e Adventurers Vault</recordname>
</link>
<name type="string">Equipment Item 1</name>
<weight type="number">0</weight>
<cost type="number">0</cost>
</equipment1>
</section001>
<section002>
...


Hi!

No particular reason DrZeuss. Perhaps having it all in one program instead of two. :)

Primarch

[valeros]

Wow, a lot to which to reply.

1) I realized after my post that I believe Tenian’s parser could put all monsters in one module. Since the parser can take a batchfile, if someone developed the batch file that appends all the different sources and only parses NPCs, then that should work. Probably someone did that already and I am just unaware of it.

2) I will look into trying to parse the “old style”monsters as life permits. But do people still really use the old monsters? And if so, don’t you have to upgrade them manually? Is being able to get just the "new" monsters good enough?

3) I did consider having mine output in a form thatTenian’s could use. In hindsight, maybe I should have done that originally. But since Tenian’s still uses tags on his output, I would still have to parse, just not as much. Also, I originally made mine specifically to make it easier to uset han Tenian’s (not saying I succeeded), so using mine followed by using his would be even worse. (No bash intended. It is a great tool. Just not super easy to use at first and especially if you want ALL the entries for a particular category.) Also, I have already created the code to output to FG, so it does not save me that.

4) As far as the XML portion:
a. Although my XML output for items is not identical to what you showed (and thank you very much for providing that!), I believe I do use the enhanced tables for equipment, armor, weapons, and magic items. Griogre helped me with that.
b. As far as combining with other modules, I intentionally make all the data (descriptions & indexes) under one XML element so that using a editor like Notepad++ you can copy that one element into an existing module. Then you copy the library entry as well and you have the data in whatever module you wanted. Not automated but fairly easy. Griogre even made a videoon how to do it.
c. Good to know about the mm3mode tag. I figured it was to distinguish “new style” from “old style”, but I was unsure for what it actually was used by FG. But it does not change the damage output to the new standards, correct?
d. I agree 100% about storing the data in HTML. So many exceptions in parsing. What a pain.

5) One thing I did not mention when referring to the demand is my own personal assumption that 50 downloads translates to about five people who actually use it. Also I did not mention the group with which I play is moving to Pathfinder and I will not be the DM. Both affect my motivation since I will not personally benefit. I did not mean anything negative by that comment. Decent point about it being a DM tool, so it benefits more than the people that use it. So again, I will look into parsing old style monsters as time permits.

6) And just to shamelessly plug myself to primarch (per "creating more things such as his item parser"), I created other tools for FG. The main ones:

a. A tool that downloads images of miniatures as tokens (now with optional border!) https://www.fantasygrounds.com/forums/showthread.php?t=17628

b. A tool that converts monster files from the Monster Builder tools (the old one and the new Silverlight one.) Not perfect, but gets most info. https://www.fantasygrounds.com/forums/showthread.php?t=17679 (https://www.fantasygrounds.com/forums/showthread.php?t=17679)

c. A tool that downloads Fortune Cards to go with the fortune card extension DrZuess created. At the end of this thread: https://www.fantasygrounds.com/forums/showthread.php?t=15352

Thank you

[Griogre]

On the old 4E monsters, I use them. Practically speaking, the differences between the stats on old and new monsters are small enough to be meaningless up until high heroic tier - the exceptions are brutes starting about 8th level.

Its so fast to update old monster stats to MM3 ones, that for me it takes more time to figure out whether the stats are old or not. The double the mod number for all non brutes and triple the mod for brutes works very well after 6-7th level.

[valeros]

2013/03/03 Updated parser:
1) Added ability to parse monsters/creatures
2) Should now build the ".mod" file automatically
3) If output directory already exists, output overrides existing files instead of deleting entire directory
4) Default file now "db.xml" instead of "client.xml" (due to monsters)

If you parse a large numbers of monsters, keep the resulting entries text file (e.g. "Monsters 5170 Entries 2013_0303.txt".) This is in case there are issues with the output of the parser, you can use that file for input when the parser is patched instead of downloading all the monsters again.

Notes about monsters:
1) Had to use "db.xml" instead of "client.xml" because otherwise FG would not show saving throws & action points line. FG would also generate an error each time a monster record was opened.

2) If a monster has a saving throw like "+2, +4 while bloodied", FG does not seem to show the line correctly. So if you see a monster and the saving throws line seems to be cut-off, that is FG, not the parser. (Examples: Querelian or Sharn)

3) If you add a "tokens" directory with a subdirectory named the same as the module to the resulting directory, those should be automatically mapped to the monster. The token name must match exactly the name of the monster, be in lower case, be a PNG file, and end with ".png". So for example, if the module was named "Monsters", then the token for "Shuffling Zombie" must be "Monsters\tokens\Monsters\shuffling zombie.png"

4) If a monster has disease, description, or revisions data as part of the entry, that data is added to the "Other" tab.

5) "Published in" data is added to the "Other" tab.

6) Indication of whether "New Style Monster" or "Original Style Monster" added to the "Other" tab.

7) Where obvious I fixed a few monsters/powers (like adding missing keywords or aura size) and converted text to common wording (e.g., variations like "while bloodied", "only while bloodied", "usable only while bloodied", "only usable while bloodied" are all converted to the same text.)

8) Right now, monsters are only listed by name or level (so not not by group, level/role, or role/level like Tenian's). If parsing all monsters, the lists are so long you need to use the search function or collapse the sections since scrolling does not work well (at least for me.)

9) There are so many variations in the data I am sure I did not parse something correctly. Please let me know what is not parsed correctly.

Some stats just for fun:
1) Currently parses 5170 monsters of which 2135 are new style and 3035 old style monsters
2) Output XML file for FG with all 5170 monsters is 497,162 lines long
3) Parser takes my computer 34 seconds to parse all monsters
4) For the old style monsters, there are approx 12,318 powers with approx 1,848 unique combinations of just the Recharge and Action fields (e.g., "Standard; At-Will", "Minor; Recharge 6", etc.)
5) The file containing the raw monster data is 129,621 lines long

[MrSavaroth]

Awesome!

Thank you very much for this :)

[askaval30]

absolutely brilliant! fast, easy to use and well crafted... worthy of all thanks and admiration!

[primarch]

Hi!

Kudos to you Valeros!!

I was wondering you ever thought adding functionality that would parse themes and related powers for use? Currently we can't do that. I think its the only thing left no one has tried.

Thanks for your excellent work!

Primarch

[valeros]

How would that work (themes that is.) There is no theme template. The description of the theme could work like backgrounds (just a formatted text in a reference text window.) But how would the powers be associated to the theme? I do not know a add a link for the associate powers to something like that.

[Zeus]

How would that work (themes that is.) There is no theme template. The description of the theme could work like backgrounds (just a formatted text in a reference text window.) But how would the powers be associated to the theme? I do not know a add a link for the associate powers to something like that.


You would need to write an extension which adds a new windowclass for 4E Themes. As you suggest you could use the Backgrounds class as a template and use the formattedtext field to link to associated powers.

You can use the <listlink> tag within formatted text fields to reference other objects in the module. e.g.



<listlink>
<link class="reference_classpowerlist" recordname="refpowerlists4eAP.bardpowers@4e Arcane Power">Bard Powers</link>
<link class="powerdesc" recordname="powerdesc4eAP.powerJinxShot@4e Arcane Power">Jinx Shot</link>
</listlink>

Where in the above example 4eAP and 4e Arcane Power are the module id and name of the module.

4E uses reference_classpowerlist and powerdesc as the windowclasses for Class Power Lists and Power sheets.

[primarch]

Hi!

Thank you DrZeuss for giving valeros that tidbit of code!

Valeros, as it stands, themes is the only thing missing to be able to have a complete scraped and parsed 4e database.

As of now, I basically use just your parser for monsters and items. Getting themes would be the proverbial cherry on top of an already great desert.

Heck, have you ever thought of making utilities for just downloading feat and powers like you did for monsters and items?

At least for me, having these "master" lists is proving to be leaps and bounds more convenient than separate in their own books.

In any even, I'd like to reiterate my thanks to DrZeuss and you for all the great stuff you've put out. Personally, I will be staying with 4e, so all these great tools are of immense long term value.

THANK YOU!

Primarch

[valeros]

Thank you for the code DrZeuss.


So I took a quick look at powers. There are a lot (9356 to be exact), but I glanced through some of them.

I got a little confused about how the XML would look for powers, so I used Tenian's parser to parse Arcane Power. I found some oddities looking at the XML output from Tenian's parser.

Example oddities:
1) Tenians parser outputs tags like special, hit, miss, effect, etc. which translate into the text with the tag name bolded at beginning of the line. (So "<effect type="string">Some effect text here</effect>" translates on screen to "Effect: Some effect text here" with the word "Effect" bolded.

2) Tenians parser outputs other special tags like firstfailedsavingthrow, hit_fortitude_, etc. that translate to headers other than the name of the tag. When I look if FG, they are translated into meaningful headers like "First Failed Saving Throw" or "Hit (Fortitude)"

3) sustain is another tag like the others but instead of the tag name being bolded, everything before the first ":" is bolded.

4) When trying to link a power, Tenian's parser output a "<linkedpowers>" tag as opposed to the "<linkedlist>" mentioned. It seemed to work.

5) I noticed what seems to me to be the same case to be parsed differently at different times. For example, Grease, Evard's Dreadful Mist, and Shard of the Endless Avalanche all parse the second power differently. Grease tries to link the power, Dreadful Mist has the second power text under the sustain line, and the second power of Endless Avalanche looks like a continuation of the main power.

Especially for the first four, do you know where this is documented or how was I supposed to know this?

[Zeus]

Thank you for the code DrZeuss.

So I took a quick look at powers. There are a lot (9356 to be exact), but I glanced through some of them.

I got a little confused about how the XML would look for powers, so I used Tenian's parser to parse Arcane Power. I found some oddities looking at the XML output from Tenian's parser.

Example oddities:
1) Tenians parser outputs tags like special, hit, miss, effect, etc. which translate into the text with the tag name bolded at beginning of the line. (So "<effect type="string">Some effect text here</effect>" translates on screen to "Effect: Some effect text here" with the word "Effect" bolded.

2) Tenians parser outputs other special tags like firstfailedsavingthrow, hit_fortitude_, etc. that translate to headers other than the name of the tag. When I look if FG, they are translated into meaningful headers like "First Failed Saving Throw" or "Hit (Fortitude)"


Yes. The powerdesc window class defines the expected fields for a power. When Jamie first wrote the parser, powers only used formattedtext fields. At the time, access the data in a formattedtext field was programatically constrained (no getValue and setValue methods) and so over time Jaime added individual string fields in the module output to make accessing data a little easier from within the ruleset.

Each power's flavour text is parsed (in its entirety) and added as the flavor tag using formatted text. In addition, the same text (minus mark-up tags) is added as a string flavour field (stringflavor). Similarly each power's descriptive text is parsed (in its entirety) and added as the shortdescription tag using formatted text. In addition, the same text (minus mark-up tags) is added as a string description field. Finally any special tags are also captured e.g. <effects> <sustain> etc. etc.

The formattedtext fields are used for displaying the power as its prettier. The string based tag data is used by the ruleset e.g. power list filtering and the keyword fields are used by the ruleset to manage the various format changes that can occur when dealing with class, item, feat powers etc. They are also used for parsing attack/damage/effect strings on the charsheet and CT.


3) sustain is another tag like the others but instead of the tag name being bolded, everything before the first ":" is bolded.

Hmm, not sure. Probably a bug. I am guessing that the Parser traps lines containing a : character and bolds all text prior to the : assumming its a name. If a line contains multiple : characters I am guessing this breaks the capture he has setup.


4) When trying to link a power, Tenian's parser output a "<linkedpowers>" tag as opposed to the "<linkedlist>" mentioned. It seemed to work.


Ah yes, each power sheet contains a small subpower list under <linkedpowers> for handling ***GRANTED*** powers. I guess you could use this to store any linked power information however the association here would be to each power and not to a theme (which I thought was what you were asking for).

Linking powers from within formattedtextfields will only work if you use the <linkedlist> tag I mentioned.


5) I noticed what seems to me to be the same case to be parsed differently at different times. For example, Grease, Evard's Dreadful Mist, and Shard of the Endless Avalanche all parse the second power differently. Grease tries to link the power, Dreadful Mist has the second power text under the sustain line, and the second power of Endless Avalanche looks like a continuation of the main power.

Especially for the first four, do you know where this is documented or how was I supposed to know this?

They all parse identically for me? Jaime's parser uses the ZPOWER tag to link to sub powers and attacks. I am guessing the input power text is not marked up correctly between all three powers, check to ensure you have the ZPOWER tag added.

Aside from the ParserDocument.txt file which is included in the Parser folder which details the expected syntax for the various different content types, I am not aware of any other documentation. You could try PMing either Tenian or EugeneZ for more information.

[valeros]

Thank you very much for the info.

So I if understand:
1) The "description" tag/field is the one used to display the data in FG. (That makes sense and matches teh fields specified in "ref_power.xml".)

2) The other special tags/fields (like effect, firstfailedsavingthrow, etc.) were used by the 4E ruleset. Do you know if they are still used by the ruleset (and therefore I would need to include them in addition to the data being in the "description" field)?

3) Same thing with the "shortdescription" field. That only seems to be used by custom powers. Do you know if I would need to include that tag/field.


Feats seem to work like powers. That is, they have special tags/fields that are duplicate of the text in the "description" tag/field. Same question as above, do you know if those duplicate fields are still needed?

Thank you

[valeros]

2013/04/07 Added Feats & Powers

The ~374 powers that have an additional power in the compendium entry (e.g., Evard's Dreadful Mist) have the additional power currently listed as part of the main power as opposed to a second separate power.

Also, for powers, there is now a required file called "Power Classes.txt". It is used to categorize the powers based on the class of the power. It is in the form "Category: Class". You can change it to make categories more to your liking. If you make a better one, please post it.

Thank you

[mattcolville]

Holy crap! Thanks a million Valeros!

[primarch]

Hi!

I second the "holy crap"! Man, Valeros your stuff rocks the house! ;)

Does this mean the only thing left for you to add are the character themes? If you achieve that then you have truly done it all! Thumbs up Valeros! :)

Edit: As I parsed and went over the book, it seems you DID get the theme powers done! Is there anything left to do? Awesome!

Primarch

[zapoqx]

Oh wow.... nicely done. You are awesome!

[Torgaard]

First off, valeros; you are the coolest dude ever. Thanks a ton for your work. Last night your parser pretty much talked me into jettisoning all the individual rulebooks I'd parsed with Tenian's Parser and just use the parses from yours. It's so much cleaner to just have like 8 books in my Library instead of 20, and it's sooooo much easier to find Powers when it's in one Library entry named "4E Powers", or Monsters in my "4E Monsters" book. Man is that sweet!

A couple things:

I parsed the Powers last night, but when I add them to a character sheet, it doesn't update the actual sheet with (a) the description, and (b) the preformatted conditions and effects for a given power (ie the little sword icon clicky on the sheet). I can click the button on the character sheet and it'll pop up the actual Power Card, but the character sheet fields themselves aren't being populated. Did I do something wrong when I parsed?
Any chance you'll support Traps in a future revision?


Thanks again, and keep up the awesome work! Tools like this are so insanely helpful for all the 4E DM's out there.

[valeros]

Thanks for the feedback.

I will have to check on the powers. My guess is that what you are seeing is related to the questions I asked above about which tags really need to be in the output. I did not output as many tags as Tenian's parser because I did not think they were used anymore. But it sounds like they are. I should be able to add them, hopefully this weekend. (So hopefully you kept the file(s) with the downloaded data so you do not have to download the entries again when I update the parser.)

Also, you can combine entries if you want into one output module. You can either specify multiple types when downloading or specify multiple files to read. That way you can have less than 8 files if you want.

I will look at traps as time permits, but I will try to fix powers first.

Thanks

[valeros]

Looks like I forgot to include the shortdescription in the powers. I fixed and the powers seem to work fine for me on the character sheet.

Torgaard, when you get a chance, could you please try the updated parser and see if this resolves the issue for you. Thank you

[Torgaard]

Sweet!

And I'm stoked to try your idea of just making one big module. That could be cool as hell. Ohhh, I know what I'll do: I'll create like a "4E Players Compendium" with like Powers, Rituals, Backgrounds, stuff like that, and then make a "4E DM's Codex" with Monsters, Terrain, etc. Awww ya.

[Torgaard]

Works! Well done. And thanks again, val.

Off to make my giant sooper modules!

[primarch]

Works! Well done. And thanks again, val.

Off to make my giant sooper modules!

Hi!

I'm gonna try this too! :D

Valeros, as I am going through the I noticed that racial and class features don't appear in the powers or feat book.

For example the deva astral resistance and astral majesty or the Battle Cleric's lore for example. I had them functional in the original books, but they don't seem to pop up on your parser. I know from previous experience that these can be "finicky" so you may not have gotten to those yet.

Also do you plan on doing conditions and skills? They are a short list, but I realized that Tenian's latest version does not parser the "nature" skill. Its real odd and I have tried a bunch of methods but it seems to "ignore" it when parsing. Earlier versions of the parser do it just fine. Also that parser doesn't really parse the skill text and tables correctly. I have done my own patch job, but its not a great solution.

Besides, skills and powers, I guess the paragon paths and epic destinies could be done as well. All these are pretty short lists and hopefully won't burden you too much.

I must tell you how grateful and thankful for all these parsers. It has made my DM'ing infinitely easier as well as cutting down the amount of books to almost nothing.

Again many thanks!

Primarch

[judesudbury]

Totally awesome Valeros. Thank you, truly
:D

[valeros]

Added Glossary entries

Skills show up under "Category: Skills"
Conditions show up under "Type: Rules Condition"

Lots of other goodness like "Type: Rules Table"

[primarch]

Added Glossary entries

Skills show up under "Category: Skills"
Conditions show up under "Type: Rules Condition"

Lots of other goodness like "Type: Rules Table"

Hi!

I bow to your coding magnificence!

I have not been this enthused with parsing for quite some time. I spent most of the afternoon putting some custom books together using your parser. It's never been this easy and good! :)

Primarch

[Invictys]

Awesome parser and super easy to use not fiddly at all like the old one!

I do have one question/issue though. How can I change the .xml type created in the module from db.xml to common.xml or even client.xml?

As it stands I seem to be unable to do this which means I can't push the module for the players to see. They all have DDI accounts but its a pain to have to look stuff up in books or on the DDI when I have all the data parsed already but they can't see it.

Am I missing some option on the parser? I know Tenians has an option for the output format, does the new one?

Either way, thanks for all the work you have put into this tool!

[mattcolville]

There's a super easy way to do this. Change the .mod to a .zip file.

Open up the .zip file.

Rename the db.xml to whichever kind you want.

Rename the .zip and .mod.

I *think* that works.

[Invictys]

Oh really? Haha I had thought about that but assumed the xml was different in some way aside from the name.

I'll give it a try and report back :)

[rbeard101]

There's a super easy way to do this. Change the .mod to a .zip file.

Open up the .zip file.

Rename the db.xml to whichever kind you want.

Rename the .zip and .mod.

I *think* that works.
Yes, This does work. I recently did it and tested it.

[meathome]

Hi, what kind of offline file does this parser except? What can you specify by add files?

[valeros]

The parser only accepts files created by the parser when you download entries.

When you dowload Compendium entries, it leaves a file name "<module name> <number of entries> Entries <date>.txt". (For example, "Monsters 5170 Entries 2013_0421.txt") That file has the downloaded entries. You can select multiple of these files if you want.

It accepts these files so that you do not have to download again from the website when you want to combine entries in different ways or if the parser changes to fix a bug or add an enhancement.

[meathome]

Any way in creating files like these? I have a lot of the compendium backed p on my machine from when i had a subscription, actually php + mysql db scrape

Im trying to get that stuff int either your or the old parser... atm i am actually trying to write a c# prog to convert the stuff into the old format.. but maybe it would be better to convert it to your format. dont know if i could post an example of what i get from my database as it is actually wotc content.

Easiest would be to renew my sub and parse it.. but strangely paypal does not work anymore for it

But at the moment i would like to get my data into fg somehow.

[adminwheel3]

Thanks for this great tool!

And I'll remove my original question as I found the answer earlier in the thread.

[meathome]

Valeros could you give me a template for those files? I managed to put together a small program with the help of the htmlagilitypack that converts my data to the format of the old parser txt files.. Sadly it doesn't like monk powers with[] in them (I simply changed all [] to() ) but it still doesn't like monk powers because they are basically 2 powers, movement and attack technique.
I think it would be much better to rewrite my program so it outputs files in the format you use for saving scrapes.

I know its much to ask, and probably not really useful for anyone but me. S I understand if you don't have the time.

I managed to get quite far with my power parsing, my problem is all powers linked to another power as a granted power dont look right.. if i go into the xml and manually give the power a recharge value(either at-will, encounter or daily) it looks right but without it all the fields overlap. This makes all linked powers unreadable... btw are the monk full disciplines supposed to parse in this way (attack technique as power with the second technique added as a granted power) ?

[zapoqx]

So I went and redownloaded the items. I got an error to pop up though in the box after everything was complete:


Mold King's Crown: Not Parsed: <H1 class=atwillpower><SPAN class=level>Level 10 Disease</SPAN>Blistering Corruption</H1>
Mold King's Crown: Not Parsed: <P class=powerstat><B>Stage 0</B>: The target recovers from the disease.<BR><B>Stage 1</B>: While affected by this stage, the target takes a -2 penalty to Fortitude.<BR><B>Stage 2</B>: While affected by this stage, the target takes a -2 penalty to Fortitude. In addition, the target regains only half the normal number of hit points from spending a healing surge, and the target’s hit point total cannot exceed its bloodied value.<BR><B>Stage 3</B>: While affected by this stage, the target suffers the effects of stage 2 and has an aura 2 (poison) that cannot be deactivated. Any creature that ends its turn in the aura loses a healing surge.<BR><B>Check</B>: At the end of each extended rest, the target makes an Endurance check if it is at stage 1 or 2.<BR><I>12 or lower</I>: The stage of the disease increases by one.<BR><I>13-17</I>: No Change<BR><I>18 or higher</I>: The stage of the disease decreases by one.<BR><BR>

[valeros]

Thank you for reporting. I add debug like that in case I miss something or something new appears in the data.

In this particular case, the data not parsed should not be a problem. It did not parse the disease associated with the item. The item should have parsed fine. (The disease can be parsed separately under diseases.) So you can ignore these particular errors for that specific item.

[valeros]

Added traps.

One issue I did not resolve yet is that traps sometimes have the same name for multiple entries. FG will only link to one even though all are parsed. For example, Glyph of Warding, Whirling Blades, and Pendulum Scythes. (I do not remember any other type having the same name for more than one entry.)

As usual, please let me know if you see anything odd.

Thanks

[meathome]

I would have a suggestion for the traps issue:
I assume the problem is that the same entry name results in duplicate database node names.
The simplest thing you could do is, have a integer counter that counts up after each parsed item, and when the parser creates the database node name simply append the counter.

Let's say you have two traps called Spike Pit after appending the counter you have database nodes called SpikeTrap94 and SpikeTrap95 (I made up the number, but counting up for every parsed item ensures that its a number unique to each entry.)

I managed to parse everything except the disease of the mold kings crown and the augmentable powers from phb3 (these do parse but loose the augment x tags)

Is there a way to show category and level for rituals? and maybe sort them by level (its one of my players big complaint since the beginning)

And for monsters the old parser had a neat function to automatically pair tokens to the monsters if a token with the same name was present in the tokens folder .. maybe with the improvement to select a standard letter token if no token is available.

Another question is it intended that the alchemical rituals don't have the items they create linked to them or , is it an error on my part?

[Invictys]

A way to automatically pair tokens would be amazing.

I find that aside from locating appropriate maps, assigning tokens to each creature takes the longest as far as setting up an encounter goes.

[valeros]

For the resulting output directory (not mod file), if you add a "tokens" directory with a subdirectory named the same as the module to the resulting directory, those should be automatically mapped to the monster. The token name must match exactly the name of the monster, be in lower case, be a PNG file, and end with ".png". So for example, if the module was named "Monsters", then the token for "Shuffling Zombie" must be "Monsters\tokens\Monsters\shuffling zombie.png"

After adding the tokens directory, use the FGPEMUtility.exe to convert resulting directory to an actual .mod file (or run the parser again which should form the .mod file if you use the same module name.)

This was to match the same way as Tenian's parser did it. That way everyone who had already created token directories could just use their existing directories (or copy the tokens.)

[valeros]

Correct, alchemical rituals don't have the items they create linked to them. Right now, nothing has a link to a different entry. The items do show up under items if you parse items.

(Closest things are powers that have a secondary power in the same compendium entry are combined into one power and diseases on monster entries are included in the Other tab of the monster.)

[valeros]

Added a new version that has level and category lists for rituals.

It seems like rituals have some sort of filtering option, but I do not know how to create the list to use it. If someone has an example of a good ritual list, I could try to use that format instead. For now, just a list of levels and categories shows up when selecting rituals (and an "All Rituals" option).

[meathome]

Noticed something strange. In the equipment list if you scroll over a price it alters the price even if mouse wheel editing is disabled or set to ctrl only. The ritual list is fine this way totally what I was looking for great work!

[primarch]

Hi!

Valeros, do you plan in including class/paragon features and epic destiny?

Is there any thing left that your parser doesn't do besides these?

Primarch

[valeros]

For items, the price can be altered because the .mod file is using a db.xml file. You will notice other entry types are also editable.

If you do not want that, in the output directory of the parser (not the .mod file) rename the db.xml to common.xml or client.xml and use the FGPEMUtility.exe to convert the directory to a .mod file.

I probably should add an option to the parser to let you pick the kind of xml file you want.

[valeros]

I will look at parsing the features that are not powers from the classes, themes, paragon paths, and epic destinies as time permits. I believe all of the powers for these should already be parsed under Powers.

The parser does not parse "Companions & Familiars" or Races. It does not parse Item Sets. It does not generally parse a second entry in a compendium entry (i.e., a disease listed in an item or an item listed in an alchemical ritual) as the second entry can be parsed under the other category.

[meathome]

Maybe I can offer some coding help, now that my little converter is finished? I mostly use C# or Java, but I am weak with regular expressions (Have a hard time with the more complex expressions)

I just noriced something strange.. When opening a ritual from the ALL rituals or by category sections everything works, in the by level lists all the right rituals are listed, but when I open one its a blank window. I opened up the db.xml file to see if I could find whats causing it.. but all the links seem to be in order... also I think it worked before, I tried rebuilding the module didnt work and I tried building a module with only rituals and had the same problem there.

[Nalvenin]

hello, how can i create mobs from an adventure module without a DDI Subscription? I have a few PDFs from D&D with the monsters in a green box. Thanks.

nalvenin

[mattcolville]

hello, how can i create mobs from an adventure module with a DDI Subscription? I have a few PDFs from D&D with the monsters in a green box. Thanks.

nalvenin

Let's imagine you wanted to create a module with all the monsters in it from The Slaying Stone, to pick a random adventure.

You're run the New Parser, which takes you to the Compendium. In the Compendium, select Creatures from the Search dropdown.

Then from the Source dropdown, select HS1 The Slaying Stone.

Then search. You'll get a list of 26 creatures. These are all the creatures in The Slaying Stone. Now you add them to the parser, and parse!

[Nalvenin]

Let's imagine you wanted to create a module with all the monsters in it from The Slaying Stone, to pick a random adventure.

You're run the New Parser, which takes you to the Compendium. In the Compendium, select Creatures from the Search dropdown.

Then from the Source dropdown, select HS1 The Slaying Stone.

Then search. You'll get a list of 26 creatures. These are all the creatures in The Slaying Stone. Now you add them to the parser, and parse!

Sorry, I meant to say without a DDI subscription. I'm not a DDI Subscriber! Can I access the compendium without subscribing?

[mattcolville]

Sorry, I meant to say without a DDI subscription. I'm not a DDI Subscriber! Can I access the compendium without subscribing?

You cannot.

[Nalvenin]

You cannot.


Ok, so again I ask, how can I create mobs from the green mob boxes in the adventure modules for use in FGII? (without having to type every single stat that is) Thanks in advance.

[mattcolville]

Sorry, I misunderstood your question. I thought you were asking how to use the Parser without a DDI subscription.

Alas, the only way to do it is by hand. The good news is, if you can copy the text of a monster ability and paste it into a new, blank, NPC entry, FGII will parse the text automatically so you can use it's bonuses to hit and damage and effects.

[primarch]

I will look at parsing the features that are not powers from the classes, themes, paragon paths, and epic destinies as time permits. I believe all of the powers for these should already be parsed under Powers.

The parser does not parse "Companions & Familiars" or Races. It does not parse Item Sets. It does not generally parse a second entry in a compendium entry (i.e., a disease listed in an item or an item listed in an alchemical ritual) as the second entry can be parsed under the other category.

Hi!

The powers are available as you mention for paragon classes and themes. My two groups find having parsed class/paragon features good to have. I do them using the old parser and made a small book covering all that.

The old parser had these parsed with the classes powers with a heading at the beginning (before listing level powers) called "features". This eliminates the need for looking up features since you can have them on your character sheet (I place them in special abilities or situational in the powers section).

Of course your parser doing it would be MUCH better. :D

Primarch

[Nalvenin]

Sorry, I misunderstood your question. I thought you were asking how to use the Parser without a DDI subscription.

Alas, the only way to do it is by hand. The good news is, if you can copy the text of a monster ability and paste it into a new, blank, NPC entry, FGII will parse the text automatically so you can use it's bonuses to hit and damage and effects.

Ok, so if I copy the green block of text and paste it into a new NPC entry, it will parse the correct info? :) (Crossing fingers)

[valeros]

Added Paragon Path features and Epic Destiny features.

Fix for rituals by level not showing ritual.

Fix for traps with same name. For this, you need to download traps again. Will not work with existing file.

For Paragon Paths & Epic Destinies, this adds only the features, not the powers. The powers can be parsed separately under powers. Also, there are a few Paragon Paths or Epic Destinies that do not parse completely (e.g., Grim Blackguard) because powers are part of a particular feature. Paragon Paths & Epic Destinies are done differently than Tenian's parser.

[meathome]

Can you send me info on how I have to update my traps data, for the change to work?

[mattcolville]

Ok, so if I copy the green block of text and paste it into a new NPC entry, it will parse the correct info? :) (Crossing fingers)

Not exactly. All the stats, you'll have to enter manually. But when you get to the text of a power, its effect, you can copy that and paste it into the right field in the NPC entry, and FG2 will 'understand' it. You'll be able to double click on +7 vs AC for instance, and it will make an attack roll, add seven, and compare it to the selected target's AC.

[valeros]

Added powers to Paragon Paths and Epic Destinies.

This also fixes the few Paragon Paths or Epic Destinies that did not parse completely (e.g., Grim Blackguard).

[primarch]

Added powers to Paragon Paths and Epic Destinies.

This also fixes the few Paragon Paths or Epic Destinies that did not parse completely (e.g., Grim Blackguard).

Hi!

Wow! It seems you get things going faster than I can test them! :D

I'm guessing class features are next?

Back to parsing more books!

Primarch

[valeros]

Are the paragon paths & epic destinies OK? I like this way better. You can copy the entry for the path to the character sheet and also copy each individual feature/power to the character sheet as desired. No need to lookup each part individually.

Probably Character Themes are next to get the theme features. Looks very similar to Paragon Paths and Epic Destinies, so hopefully fairly easy.

And fixing the "Augment" issue with powers. And would also like to handle power entries that have secondary powers differently instead of combining into one entry.

[valeros]

Added Character Themes

Added option for database type (db.xml, client.xml, common.xml)

[primarch]

Are the paragon paths & epic destinies OK? I like this way better. You can copy the entry for the path to the character sheet and also copy each individual feature/power to the character sheet as desired. No need to lookup each part individually.

Probably Character Themes are next to get the theme features. Looks very similar to Paragon Paths and Epic Destinies, so hopefully fairly easy.

And fixing the "Augment" issue with powers. And would also like to handle power entries that have secondary powers differently instead of combining into one entry.

Hi!

Valeros, the paragon and epic destinies are great. Being able to select one and have all features right there to port over as links is a great feature and very easy to use!

I also like that you used the same format for the character themes, since my group will be picking some out and this eases the process considerably.

I assume this is the format you will use on the last two big categories remaining, such as Class (which will have the class features) and race (which includes racial features)?

If you tackle these two the same format you used for paragon paths and themes would be great.

By my count only Classes and Race remain to add to your parser (Also companions, but this one is not as used as perhaps Classes and Race.

Great work Valeros!

Primarch

[valeros]

Added Classes and Races.

Changed backgrounds to be like feats (so they can be added to a character sheet.)

Fixed "Augment" keyword issue with powers.

Fixed flavor text on Themes, Paragon Paths, and Epic Destinies.


Note: three different approaches taken for Themes vs. Races vs. Classes
1) Themes, Paragon Paths, and Epic Destines: Parsed out features as links with no feature text in main entry.
2) Races: Parsed out features with links and left feature text in main entry.
3) Classes: Did not parse out features but left feature text in main entry.

(In all cases, the powers are broken out as links with no power text in the main entry.)

Here was my thinking:
At first (Themes, Paragon Paths, and Epic Destines) were broken out with features working like feats/powers because that was requested and was how Tenian's parser did it.

Then as I parsed Races into features, I realized most features were very short. Since the text of the feature was very short, it was more painful to open the parsed feature power than to just see the feature text in the entry. But since I had already parsed, I left the parsed feature powers in the entry, but also left the feature text in the entry.

As I parsed Classes, since the Class power containing all the text (and powers) about the class in contained in the entry, and thus is available for easy perusal, why break about the features as feature powers? It does not seem to add any additional value. Previously with Tenian's there was not an overall entry containing all the info about a class (to my knowledge anyway) so you needed each power and feature separately. This way you do not.

[valeros]

Added "easy" buttons for making a Player's module or a Dungeon Master's module

[primarch]

Hi!

Kudos to you sir! With these last additions, I doubt I will ever use the old parser again!

I have reduced the contents of the compendium to about 6 books using your parser. My players not only love it, but loading times are significantly reduced using a few books compared to the nearly 20 I had open at one time.

Pure efficiency!

Primarch

[Nalvenin]

Hello, what is the format of a file for the parser to recognize it? IE: npc.xml

[valeros]

The parser only accepts files created by the parser when you download entries.

When you dowload Compendium entries, it leaves a file name "<module name> <number of entries> Entries <date>.txt". (For example, "Monsters 5170 Entries 2013_0421.txt") That file has the downloaded entries. You can select multiple of these files if you want.

It accepts these files so that you do not have to download again from the website when you want to combine entries in different ways or if the parser changes to fix a bug or add an enhancement.

[Nalvenin]

The parser only accepts files created by the parser when you download entries.

When you dowload Compendium entries, it leaves a file name "<module name> <number of entries> Entries <date>.txt". (For example, "Monsters 5170 Entries 2013_0421.txt") That file has the downloaded entries. You can select multiple of these files if you want.

It accepts these files so that you do not have to download again from the website when you want to combine entries in different ways or if the parser changes to fix a bug or add an enhancement.

Thank you for the clarification. :)

[Seraphim]

I remember using the old parser and remembering how clunky it was. This new one looks really nice and easy to use.

Perhaps you fine gents can help steer me in what I'm doing wrong. Unfortunately, mine seems to crash every time I get so far into a scan. I'm on Win 8 pro, 64 bit.

1. I click "Add Dungeon Master Entries"
2. It gathers the entries. Takes a minute or two. This is a screenshot of the final result:

https://img.photobucket.com/albums/v662/Ivno/Part1.jpg

3. So far so good. I enter my email, and password of DDI, click "Download and Parse Selected Entries" and we're off.

4. It seems to scrape quite a few files, probably for about half an hour to an hour and then it always crashes. The program freezes and I get a, "Parser has stopped working." error.

I've tried using just 1 browser, 5 browsers, and ten browsers. Not really sure where the issue lies or what steps I should take from here on. I'm going to try and run it again and if it errors up I'll take another screenshot of what happens. Cheers. : )

[valeros]

1) Keep the data file that is produced. It should be named something close to "4E Dungeon Master's Guide 6530 Entries 2013_0521.txt"

2) I would bet that the issue is something about Windows 8 or IE 10. The HTML pulled from the compendium probably does not look the same as when using WIndows 7 or IE 9 or earlier. The parser is really picky about the HTML because I wanted to be sure I parsed everything because the entries are very inconsistent. If you are willing, please email me ([email protected]) the file mentioned above and I will see if it is an issue with the HTML being different. If so, I may be able to adjust the parser to handle depending on how different it is. But if you have another computer with Windows 7 or IE 9 or earlier, I would try it on that computer.

3) Also, if you are not even getting the file, then try doing each type (e.g., poison, races, etc.) individually and see if that works. You can always combine the results later with the file capability. See if something small like poisons work.

[Nalvenin]

1) Keep the data file that is produced. It should be named something close to "4E Dungeon Master's Guide 6530 Entries 2013_0521.txt"

2) I would bet that the issue is something about Windows 8 or IE 10. The HTML pulled from the compendium probably does not look the same as when using WIndows 7 or IE 9 or earlier. The parser is really picky about the HTML because I wanted to be sure I parsed everything because the entries are very inconsistent. If you are willing, please email me ([email protected]) the file mentioned above and I will see if it is an issue with the HTML being different. If so, I may be able to adjust the parser to handle depending on how different it is. But if you have another computer with Windows 7 or IE 9 or earlier, I would try it on that computer.

3) Also, if you are not even getting the file, then try doing each type (e.g., poison, races, etc.) individually and see if that works. You can always combine the results later with the file capability. See if something small like poisons work.

I was going to add to try doing just one selection, like a magazine or monster vaul and see if that works... this is a BIG file with everything.... since it's going to take long, any little interruptions in communication can cause it to stall... :)

[Seraphim]

1) Keep the data file that is produced. It should be named something close to "4E Dungeon Master's Guide 6530 Entries 2013_0521.txt"

2) I would bet that the issue is something about Windows 8 or IE 10. The HTML pulled from the compendium probably does not look the same as when using WIndows 7 or IE 9 or earlier. The parser is really picky about the HTML because I wanted to be sure I parsed everything because the entries are very inconsistent. If you are willing, please email me ([email protected]) the file mentioned above and I will see if it is an issue with the HTML being different. If so, I may be able to adjust the parser to handle depending on how different it is. But if you have another computer with Windows 7 or IE 9 or earlier, I would try it on that computer.

3) Also, if you are not even getting the file, then try doing each type (e.g., poison, races, etc.) individually and see if that works. You can always combine the results later with the file capability. See if something small like poisons work.

Actually you are right. It was some sort of conflict with either my windows or my browser. I've tested it on another PC and got the download finished and installed. Thanks though Valerous. This parser is simply amazing for making these less complex.

[valeros]

Added new version with minor updates:
1) Mildly improved GUI look (to improve understanding)
2) Monster indexes now include options for seeing old or new monsters only. (I usually only care about new monsters.)
3) Deities work like feats to show info better and so they can be added to character sheet.
4) Traps now have alphabetic index.
5) Classes & Races indexes now just list name.
6) Features use the feature name in the description instead of a generic name (e.g., "Arcane Sword Feature" instead of "Epic Destiny Feature")

I did a lot of refactoring of the underlying code, so I may have inadvertantly added new bugs. Please let me know if you see anything broken. Thank you

[Griogre]

Looking good Valeros. Out of curiosity, if you don't like the older monsters have you considered an option to update them during the parse? It would seem pretty trivial to do during the scrape.

[primarch]

Looking good Valeros. Out of curiosity, if you don't like the older monsters have you considered an option to update them during the parse? It would seem pretty trivial to do during the scrape.

Hi!

Now THAT is a great idea. I tend to update them manually when I make an adventure now. But to have them updated already would be a HUGE improvement.

What you think Valeros?

Primarch

[primarch]

Hi!

Valeros, I may have discovered a potential "bug" on the last parser release.

It seems that in "items" regardless if the parse is doing the items alone or as part of a download/parser with several other things the entries specifically pertaining to IMPLEMENTS (heroic, paragon and epic) the lists are cut short.

For example the total item count for items is 10040. Implements have three sections:

Magic-implement-heroic (685)
Magic-implement-paragon (1044)
Magic implement-epic (1127)

Under magic-implement-heroic (685) The last entry is "Staff of the Evader +1". There are no entries after this thus no heroic implements that appear later in the alphabetical order (like all the wands for example).

The same happens under the paragon implements which the last entry is "roaring bear totem +4", no items starting "s" or later appear.

Same under epic implements where the last entry is "Panther totem +6" no items letters "r" or later appear.

The truncation of the item list appear to happen exclusively under the implement headings. No other item heading (armors, weapons, etc) shows this.

Note I CAN SEE the MISSING implement items in the raw downloaded text file. I did this 3 times, doing items on their own and in conjunction with other entries and I see the same truncation of the list for implements always ending in the same entries noted above, although they are present in the raw files.

I also downloaded parsed them in different formats like common, client, db and the same problem is seen.

Note that using an earlier parser this DOES NOT occur. It appears to have been introduced only in the latest one.

I have the completed modules as well as the raw files if you'd like me to send them to you for troubleshooting.

Let me know how I can be of assistance to you in solving this.

Thanks!

Primarch

[valeros]

I will check into this.

But at first it looks to me like the magic item list is limiting the number of entries to the first 500. I also see it on items other than implements (try anything over 500 entries). On my screen, you can tell that it seems to think there are more entries in the list because the icon of the last entry is not shown fully.

Also, I tried parsing with older parsers (from 3/3/2013 and 11/12/2012) and got the same issue.

[valeros]

I checked the 4E.pak file. The maximum height for the "reference_magicitemtablegroup" in "ref_magicitem_lists.xml" is set to 10000. I checked the one for monsters and found it was set to 100000. So I changed the value to 100000 and that fixed the issue.

So I would recommend that a patch request be made, though I do not know how that is done.

In the mean time you can manually change your 4E.pak file.

<windowclass name="reference_magicitemtablegroup">
....
<maximum>
<width>600</width>
<height>100000</height>

[primarch]

I checked the 4E.pak file. The maximum height for the "reference_magicitemtablegroup" in "ref_magicitem_lists.xml" is set to 10000. I checked the one for monsters and found it was set to 100000. So I changed the value to 100000 and that fixed the issue.

So I would recommend that a patch request be made, though I do not know how that is done.

In the mean time you can manually change your 4E.pak file.

<windowclass name="reference_magicitemtablegroup">
....
<maximum>
<width>600</width>
<height>100000</height>

Hi!

Yup, that fixed it for sure. :)

I suggest forwarding this request to ddavision or moon_wizard. You can PM them, thier usernames and link to contact them are easily found in the "city hall" forum.

It would seem a very easy to rectify this. :)

Primarch

[Seraphim]

Kind of odd. All base armor in my library have no item cost except for the items in the first players handbook. Some other random items (Like a couple of weapons) lack stats as well. Maybe this isn't a parser issue though.

[Griogre]

Just to make sure we are on the same page, masterwork armor is only used with magic armor of a certain plus or higher so it is never bought stand alone - the price of the masterwork armor is included in the base magic item cost. Some examples of these armors are Feyweave, Starweave, Githweave, ect. AV has all masterwork armor, while all the base armors are in the PH, HotFL, HotFK, and some new ones in Mordenkainens.

[Griogre]

I went to updated my Dark Sun Campaign setting book and I noticed a problem with the parser Source Drop Down. Normally for a campaign book I would select Backgrounds under Search then Dark Sun Campaign Setting under Source. The problem I found is that under Source the drop down list is cut off at "-" and just shows the top six or seven lines. This is true for Backgrounds, Character Themes, Classes, and so on; in fact, it seems to be cut off for anything but Creatures, Glossary, Items and Powers.

https://i105.photobucket.com/albums/m222/Griogre/SourceShot_zps1aef72b1.png

[valeros]

That is the way the Compendium appears in Internet Explorer. The application is using IE for that window. So if you launch IE and go to the Compendium, that is what the menu looks like. So I do not believe it is an issue with this application, but with how IE renders the page. Nothing I can do about that.

Chrome seems to render the menu correctly. And I have not tried this, so have absolutely no idea if it will actually work. But if you load the Chrome overlay on top of IE, maybe then IE (and thus this application) would work???

[Griogre]

Yeah in just the compendium it sorts right with chrome. I normally use chrome so I didn't know about IE that's why I thought it was weird. :p When I get a chance I'll research the overlay more and see what happens.

[primarch]

Hi!

Valeros, with 5e playtesting wrapping up there is a lot of talk about the "demise" of DDI.

Although I have already future proofed myself with your parser, was there any data left in the compendium your parser couldn't get?

Thanks!

Primarch

[valeros]

As far as I know, it should work with everything but "Companions & Familiars". I never did those entries.

[primarch]

As far as I know, it should work with everything but "Companions & Familiars". I never did those entries.

Hi!

Are you planning to include those in the parser, or is the parser "done".

Thanks!

Primarch

[valeros]

I do not really have the desire to do those entries, so basically "done". But willing to try to fix bugs or potentially modify if a new version of FG or the ruleset needs output in a different format.

[primarch]

I do not really have the desire to do those entries, so basically "done". But willing to try to fix bugs or potentially modify if a new version of FG or the ruleset needs output in a different format.

Hi!

No problem. I just wanted to know since I have multiple backed up copies of the entire compendium and the raw data to manipulate I don't really need a DDI subscription anymore.

Thanks!

Primarch

[Griogre]

I just parsed all the items and I thought I'd mention there is one item that throws an error, on the Mold King's Crown.

[Xorn]

So... I finally used the parser for the first time. How incredibly amazing this thing is.

So... if I want to strip out anything NOT in the 4E Essentials books, can I just go through the db.xml file and delete each node I don't want? (I understand there's a LOT of them.)

[Griogre]

Yeah, that would work. Technically you really only have to update the indexes but that's most of the work anyway since there are multiple index entries per data point. Its unfortunate the Select Compendium Entries doesn't work right. It's a bug in their code for Internet Explorer where some entries don't appear after a dash. I've complained to WotC, and they might fix it if enough people complained - just because I assume they are going to use the same DDI framework for 5E. I'm not sure if you noticed it but the Other tab has page numbers and source(s) so if you trust you players not to mess up you could just tell them to check.

[Xorn]

Did you ever figure out an IE overlay addon to get the Source filter to populate correctly? I wish it was possible to force the Parser to use Chrome. :(

[Xorn]

All finished! Wasn't too bad. Items, Powers, and Creatures could all be filtered to just scrape the Essentials stuff, so the only really big list I had to edit was Feats. Collapsed all the nodes and just went to town clearing them out, then made the index match. After that I took my Monster Vault tokens and scanned them (never punched them out of the cardstock) then made a set of tokens with TokenTool. Finally I edited the token entries for all 305 creatures (since so many of them use the same token). I broke the DM stuff and MM stuff into two different modules though.

Anyway, all finished building my Essentials modules! Thanks again for the amazing utility. I've done a full 4E PHB scrape, 4E DMG scrape, and then I also did individual scrapes for each category as well, just in case the 4E DDI dies sooner rather than later.

[monkinmeditation]

Hi All,

First, a big thank you to Valeros and everyone who has been involved with the parser; it's amazingly useful. I've been having trouble getting the Powers to parse. I tried using the player entries, as well as trying to scrape and parse the powers separately. The scraping seems to go well, but all the powers fail to parse. All of the other entries seemed to work without issue. Has anyone else encountered this issue, and if so, can you share how you fixed it? Thanks!

[Griogre]

I ran the parser last night to check after seeing monkinmeditiation's post and there are a ton of fail to parse entries. I presume 1 for each power. One thing I noticed when I ran it last night, that when gathering powers the number found were a lot more than when I ran it last: i.e. 7 thousand plus vs. 4,600ish (I only parse rulebook powers) and obviously there hasn't been any new rulebooks published since I last ran that module in August.

[monkinmeditation]

Yeah, when I scraped all of the powers, I got something like 9,380 entries, and every single one of them failed to parse. All of the other categories (Player Entries and DM Entries) seemed to work just fine otherwise and I was able to load the modules without incident. I have two copies of the powers .txt files (if that matters).

Thanks, Griogre for checking on this.

[valeros]

Sorry powers stopped working. Unfortunately, I have already let my DDI subscription lapse since there did not seem to be any updates. So there is not a good way for me to troubleshoot the problem. The parser does rely heavily on the HTML, so if that changed, it could cause a problem. Also, the parser was written for Windows 7. If you are running Windows 8 and a newer Internet Explorer, it is unlikely but possible that is the cause.

[monkinmeditation]

Hi Valeros. Thanks for replying. I am using Windows 7, for what it's worth. Being respectful of your time, do you have any suggestions for what I might be able to do? My DDI subscription lasts for another 2 weeks or so.

Thanks again.

[valeros]

If you email me the powers txt file (or 3+ powers copied out of the file), I will see if I can quickly fix the issue. Since all powers are failing, hopefully there is a single easy fix.

[Griogre]

I'm running Win7 as well. Hopefully its an easy fix.

[monkinmeditation]

Hi Valeros,

Thank you very much for your kind offer to help. I'll send you the file as soon as I get back to my computer.


If you email me the powers txt file (or 3+ powers copied out of the file), I will see if I can quickly fix the issue. Since all powers are failing, hopefully there is a single easy fix.

[monkinmeditation]

Thanks again to Valeros; he kindly took the time to look at my file. It turns out it was user error :o; I had moved the parser from its original folder and left the Power Classes.txt file in the folder. Once I moved the scraped text file into the same folder as the parser and Power Classes.txt file, it worked perfectly. Sorry for the false alarm.

[Griogre]

Yeah, and I was an idiot. I found I had deleted the power file while I was cleaning up the modules in the folder. :p

[Griogre]

With the DDI for 4E's future becoming murky I started preping to do the final parses after the last Dungeon Magazine for 4E is published in December. I noticed a problem with monsters not having the leader keyword, ie a Frost Goblin Hexer which is a "Level 3 Elite Controller (Leader)" npc entry only lists him as a "Level 3 Elite Controller." Anyone else have this problem?

[valeros]

Looks like a bug. I attached a new version on the first post. I left the old one just in case I introduced some other bugs because between the last release and this fix I had changed a lot of code doing code cleanup.

[Griogre]

I got a chance to try the new parser and got an error about using an abstract constructor. I've attached a screen shot so you can see the selections but it was just an all items for the DM test run:

https://i105.photobucket.com/albums/m222/Griogre/4EParse_zpsdeb4675a.png

Edit: After glancing at the stack trace I decided to try running the parser with only one browser and without entering the e-mail and password for the my DDI account in the Parser Interface. The parser then ran, but one browser is pretty slow. :p

Edit 2: This did complete with the following item errors. I remember the Mold King's Crown didn't work last time but the others are new:

Wintersnap Armor: Not Parsed: [<H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura) Daily (Minor Action)</H2>] in ⫪㐽魏뉛袏䍺੐
Wintersnap Armor: Not Parsed: <H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura) Daily (Minor Action)</H2>
Brassburn Armor: Not Parsed: [<H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Attack Power (Aura, Fire) Daily (Immediate Reaction)</H2>] in ⫪㐽魏뉛袏䍺੐
Brassburn Armor: Not Parsed: <H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Attack Power (Aura, Fire) Daily (Immediate Reaction)</H2>
Armor of Scintillating Colors: Not Parsed: [<H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura, Charm) Daily (Minor Action)</H2>] in ⫪㐽魏뉛袏䍺੐
Armor of Scintillating Colors: Not Parsed: <H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura, Charm) Daily (Minor Action)</H2>
Mold King's Crown: Not Parsed: <H1 class=atwillpower><SPAN class=level>Level 10 Disease</SPAN>Blistering Corruption</H1>
Mold King's Crown: Not Parsed: <P class=powerstat><B>Stage 0</B>: The target recovers from the disease.<BR><B>Stage 1</B>: While affected by this stage, the target takes a -2 penalty to Fortitude.<BR><B>Stage 2</B>: While affected by this stage, the target takes a -2 penalty to Fortitude. In addition, the target regains only half the normal number of hit points from spending a healing surge, and the target’s hit point total cannot exceed its bloodied value.<BR><B>Stage 3</B>: While affected by this stage, the target suffers the effects of stage 2 and has an aura 2 (poison) that cannot be deactivated. Any creature that ends its turn in the aura loses a healing surge.<BR><B>Check</B>: At the end of each extended rest, the target makes an Endurance check if it is at stage 1 or 2.<BR><I>12 or lower</I>: The stage of the disease increases by one.<BR><I>13-17</I>: No Change<BR><I>18 or higher</I>: The stage of the disease decreases by one.<BR><BR>
3728 Total Entries
10495 Total Resulting Parsed Entries
done

[Griogre]

I have a request for the parser, could you please set all items in modules to identified? Currently all the items are marked as unidentified (or more correctly because there isn't an identification element - the ruleset defaults to unidentified). This means players with lite versions of FG can never see the magic item descriptions in modules and while they can see normal equipment as soon as they drag and drop it into their character's inventory it becomes an unidentified object.

This is happening because I use the "Item: Identification On" setting. Personally, I can care less about identified/unidentified status and I would normally just set it off - but when you do so it shuts off the note field on the item, which I do find very useful.

The element for setting identification is below with the 1 meaning true.


<isidentified type="number">1</isidentified>


While this can be fixed with a selected lines mass find an replace, I'm probably not the only DM who has player's complaining. :(

[Zeus]

While this can be fixed with a selected lines mass find an replace, I'm probably not the only DM who has player's complaining. :(

Another option would be to add another game option to 4E that forces all items as identified by default. This should be achievable by overriding CoreRPG/4E's default item window class so that it always sets isidentified to 1. That way you will always have access to the notes. Or change the 4E item campaign/reference window classes so that notes are always displayed. I can add it to the to do pile if you think it would be useful option moving forward.

[Griogre]

Dr. Zeus there is an option that forces all items to be identified by default. Unfortunately it caused the note section on the magic item window to not display. Personally, I think it would be better to make the note field always display. I've found it useful to note which character's wishlist the item is from and the encounter it was found in. I've also found the reference book and page number the parser puts in note field for each item to be nice bonus.

If notes always on was slipped into the 4E item campaign/reference window classes in the future, that would be great.

[valeros]

Added new version (2013_1116) 1) Added all items in modules are identified. 2) Mold King's Crown - only the disease entry added to the Crown entry is not parsed with the Crown entry. The Crown item itself is parsed. The disease can be parsed under diseases. 3) Sorry, for the crash, not sure if I fixed the root issue. I was able to reproduce the crash, and this new version does not crash for me so hopefully it is fixed. Also, the new version should give a little bit better info on a crash if it crashes. 4) Sorry, not sure on the armor items either. I am not getting those errors with either the old or new version so I cannot reproduce the error. Maybe those three items have changes since the last time I downloaded entries. And I no longer have my DDI subscription to check the discrepancy.

[br0k30n3]

I have seen some video and some screen shots where people have been able to add individual modules for each rule book. I was wondering is there was a tutorial around that would demonstrate how to parse from a specific source. I played around with DDI Compendium was looking to extract all classes from that source the Compendium only shows two classes.

EDIT:

Ok so I see what may be my issue , so I click on select Compendium Entries. Now on the integrated browser it has the Compendium , for some reason under source the drop down list does not show anything beyond the -


Update:
The problem with the drop down menu appears to be a issue with Internet Explorer and CSS

[valeros]

I put a new version (4eParser_2014_0201.zip) on the first post. There is now a button to the right of "Add Entries" called "Add All Books to Source". If you have already selected "Select Compendium Entries" and have waited for the page to finish loading, you can then use this button to add all the Source Book options to the "Source" filter. This adds all books, not just the relevant ones. Also, if you use this, you must have the "All Source Books.txt" file in the same directory as the parser (just like it requires the "Power Classes.txt" for powers.) You only have to use this button once, not each time you go to a new tab.

[Griogre]

Thanks Valeros, I'll try this.

[br0k30n3]

I put a new version (4eParser_2014_0201.zip) on the first post. There is now a button to the right of "Add Entries" called "Add All Books to Source". If you have already selected "Select Compendium Entries" and have waited for the page to finish loading, you can then use this button to add all the Source Book options to the "Source" filter. This adds all books, not just the relevant ones. Also, if you use this, you must have the "All Source Books.txt" file in the same directory as the parser (just like it requires the "Power Classes.txt" for powers.) You only have to use this button once, not each time you go to a new tab.

This is awesome , Thanks Sir

[spike1288]

Hi i have used the new updated parser on the players hand book with nps at all, when it came to parsing the dungeon masters stuff i got this error when i loaded up fantasy grounds so it wont load in the modules section

Database Error: A XML parse error occurred processing file 4E Dungeon Master's Guide:db.xml - Error on line 306486: Error reading end tag.

if someone could please help me fix this it would be awesome thank you for your time :)
Spike

[LootaBox]

Hey! There seems to be a bug with the new version, that causes a whole bunch of source book filters to appear, and none of them actually work as far as I can tell. Happens if you search for whatever, click something, so you open up a popup with login confirmation, and then detailed information about whatever you just clicked on. When you close that the filter is messed up.

It doesn't really bother me, since I can just open the compendium on a regular browser at the same time. And thanks, really, for an awesome parser!

This is a bit off topic, but as far as I know varelos has no plans to incorporate familiar, companions or summons parsing to his parser, and the other parser doesn't support companions or summons either. Is there a parser that does?

[br0k30n3]

Hi i have used the new updated parser on the players hand book with nps at all, when it came to parsing the dungeon masters stuff i got this error when i loaded up fantasy grounds so it wont load in the modules section

Database Error: A XML parse error occurred processing file 4E Dungeon Master's Guide:db.xml - Error on line 306486: Error reading end tag.

if someone could please help me fix this it would be awesome thank you for your time :)
Spike

I got the same issue, I have even attempted to package the Module manually , getting the same result

[valeros]

So I do not have my DDI subscription anymore, so hard for me to troubleshoot. To my knowledge, the only change to the most recent parser was for the "Source" option. Nothing else should have changed. And to my knowledge, no data in the Compendium should be changing since I do not think WoTC is updating that anymore. If you parse the downloaded data file (i.e., do not download the data again) with the oldest parser still posted, does the error still happen? If so, that probably means something changed in the data or the data downloaded incorrectly (although that does not seem likely since multiple people are having the issue.) If not, then it is the parser. But if that is the case, you should now have a working module. Another option is to try one of the older parsers and download all the data again. But if the previous step did not work, this is not likely to work either.

[br0k30n3]

So I do not have my DDI subscription anymore, so hard for me to troubleshoot. To my knowledge, the only change to the most recent parser was for the "Source" option. Nothing else should have changed. And to my knowledge, no data in the Compendium should be changing since I do not think WoTC is updating that anymore. If you parse the downloaded data file (i.e., do not download the data again) with the oldest parser still posted, does the error still happen? If so, that probably means something changed in the data or the data downloaded incorrectly (although that does not seem likely since multiple people are having the issue.) If not, then it is the parser. But if that is the case, you should now have a working module. Another option is to try one of the older parsers and download all the data again. But if the previous step did not work, this is not likely to work either.

I looked at the XML file that it made, I didn't see anything that looked off, Possibly because to the size or number of entries ? I'll try to use it and do it for each course separately. I was having issues prior to the new update with the DM entries, just didn't dive into it with any depth until now.

[xKrisx]

So every time I try and scrape the creatures info using the Parser I get a 'Database Error: A XML parse error occurred processing file Monsters:db.xml - Error on line 239812: Error reading tag.'

What is going on and how is that fixed? I can parse everything in the DM compendium except the Creature entries. Any help would be awesome.

[Trenloe]

So every time I try and scrape the creatures info using the Parser I get a 'Database Error: A XML parse error occurred processing file Monsters:db.xml - Error on line 239812: Error reading tag.'

What is going on and how is that fixed? I can parse everything in the DM compendium except the Creature entries. Any help would be awesome.
There is an XML format error in the monsters db.xml file - this will be caused by some issue in the data downloaded from the DDI website. It can sometimes be possible to identify what the error is and manually modify the db.xml file and then run this through the parser again (don't re-scrape from the website).

Open the file in an XML enabled editor (Notepad++ for example - which is free) or a XML validator website such as this: https://www.w3schools.com/xml/xml_validator.asp Although the website may not be able to process such a large file. These may help you to identify where the XML error is and fix the issue. The problem is that sometimes the line where the error is reported is not where the error occurs...

If that doesn't work for you, try posting some of the data here - perhaps 20 or so lines either side of line 239812 to see if it is something obvious. If you're not sure where that line is, open the file in Notepad++ (https://notepad-plus-plus.org/) which will show you the line numbers.

[xKrisx]

When I open the Monsters 5309 Entries 2014_0206 in Notepad++ there is no 239812 code line to look at it only going up to like 138k

[xKrisx]

So I am still lost

[Trenloe]

So I am still lost
Where, exactly, are you seeing the error?

[br0k30n3]

Database Error: A XML parse error occurred processing file Monsters:db.xml - Error on line 239812: Error reading end tag.
-------------------------------------------------------------------------------------------

<MooncalfRogue58161>
<name type="string">Mooncalf Rogue</name>
<levelrole type="string">Level 15 Solo Soldier</levelrole>
<type type="string">Huge aberrant magical beast</type>
<xp type="number">6000</xp>
<hp type="string">620; Bloodied 310</hp>
<init type="number">12</init>
<ac type="number">31</ac>
<fortitude type="number">30</fortitude>
<reflex type="number">27</reflex>
<will type="number">27</will>
<speed type="string">6, fly 8</speed>
<senses type="string">Darkvision</senses>
<perceptionval type="number">16</perceptionval>
<saves type="string">+5</saves>
<actionpoints type="string">2</actionpoints>
<specialdefenses type="string">Resist 10 lightning, 10 thunder</specialdefenses>
<alignment type="string">Evil</alignment>
<languages type="string">understands Deep Speech, telepathy 20</languages>
<skills type="string">Stealth +15</skills>
<strength type="number">25</strength>
<constitution type="number">27</constitution>
<dexterity type="number">16</dexterity>
<intelligence type="number">21</intelligence>
<wisdom type="number">19</wisdom>
<charisma type="number">21</charisma>
<text type="formattedtext"><p>Published in Monster Vault: Threats to the Nentir Vale, page(s) 81.</p><p>New Style Monster</p></text>
<token type="token">tokens/Monsters/mooncalf rogue.png@Monsters</token>
<powers>
<id-00001>
<name type="string">Psychic Veil</name>
<shortdescription type="string">The mooncalf is invisible to creatures more than 6 squares away from it.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00001>
<id-00002>
<name type="string">Superior Brain</name>
<shortdescription type="string">Any dazing or stunning effects on the mooncalf end at the start of its turn.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00002>
<id-00003>
<name type="string">Tentacle Snare</name>
<shortdescription type="string">When the mooncalf moves, all creatures grabbed by it are pulled with it.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00003>
<id-00004>
<name type="string">Tentacle Slam</name>
<action type="string">Standard</action>
<recharge type="string">At-Will </recharge>
<icon type="string">circleM</icon>
<powertype type="string">m</powertype>
<shortdescription type="string">Attack: Melee 6 (one creature not grabbed by the mooncalf); +20 vs. AC\rHit: 2d12 + 10 damage, and the target falls prone.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00004>
<id-00005>
<name type="string">Tentacle Grab</name>
<action type="string">Standard</action>
<recharge type="string">At-Will </recharge>
<icon type="string">M</icon>
<powertype type="string">M</powertype>
<shortdescription type="string">Attack: Melee 6 (one Huge or smaller creature not grabbed by the mooncalf); +18 vs. Reflex\rHit: 1d12 + 7 damage, the mooncalf pulls the target up to 5 squares, and the target is grabbed (escape DC 22). The target takes ongoing 10 damage until the grab ends.\rSpecial: The mooncalf can grab up to two creatures at once. It can release a grabbed creature as a free action.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00005>
<id-00006>
<name type="string">Vile Bite</name>
<action type="string">Standard</action>
<recharge type="string">At-Will </recharge>
<icon type="string">M</icon>
<powertype type="string">M</powertype>
<shortdescription type="string">Attack: Melee 1 (one creature grabbed by the mooncalf); +20 vs. AC\rHit: 2d10 + 12 damage, and the target cannot spend healing surges until the start of the mooncalf's next turn.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00006>
<id-00007>
<name type="string">Multiattack</name>
<action type="string">Standard</action>
<recharge type="string">At-Will </recharge>
<icon type="string">X</icon>
<powertype type="string">X</powertype>
<shortdescription type="string">Effect: The mooncalf makes three melee attacks, each against a different target.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00007>
<id-00008>
<name type="string"><P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P></name> this is line 239812
<action type="string">Minor</action>
<recharge type="string"><P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P> <P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P></recharge>
<keywords type="string"><P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P></keywords>
<icon type="string">X</icon>
<powertype type="string">X</powertype>
<shortdescription type="string">Effect: The mooncalf activates an aura 3 that lasts until the end of the mooncalf's next turn. Creatures in the aura cannot shift. Any Medium or smaller creature that ends its turn in the aura falls prone.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00008>
<id-00009>
<name type="string">Tentacle Flurry</name>
<action type="string">Triggered</action>
<recharge type="string">At-Will </recharge>
<icon type="string">B</icon>
<powertype type="string">C</powertype>
<shortdescription type="string">Trigger: An enemy escapes the mooncalf's tentacle grab.\rAttack (Immediate Reaction): Close burst 1 (enemies in the burst); +20 vs. AC\rHit: 2d12 + 10 damage.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00009>
</powers>
</MooncalfRogue58161>

[Trenloe]

Database Error: A XML parse error occurred processing file Monsters:db.xml - Error on line 239812: Error reading end tag.
-------------------------------------------------------------------------------------------

<MooncalfRogue58161>...

Ah, OK - it is in the generated FG XML file and the error is appearing in Fantasy Grounds, not the original parsed file (although the error will come from that file) during the parsing process.

There are 2 different sets of unclosed XML tags in the <id-00008> section:

<aura align=top>
<x>

Remove these completely so that you have the following for <id-00008>:

<id-00008>
<name type="string"><P class="flavor alt"> <B>Windwrack</B> Recharge 5-6</P></name>
<action type="string">Minor</action>
<recharge type="string"><P class="flavor alt"> <B>Windwrack</B> Recharge 5-6</P> <P class="flavor alt"> <B>Windwrack</B> Recharge 5-6</P></recharge>
<keywords type="string"><P class="flavor alt"><B>Windwrack</B> Recharge 5-6</P></keywords>
<icon type="string">X</icon>
<powertype type="string">X</powertype>
<shortdescription type="string">Effect: The mooncalf activates an aura 3 that lasts until the end of the mooncalf's next turn. Creatures in the aura cannot shift. Any Medium or smaller creature that ends its turn in the aura falls prone.</shortdescription>
<mm3mode type="string">1</mm3mode>
</id-00008>
This should allow FG to read the Monster db.xml entry.

[xKrisx]

Where, exactly, are you seeing the error?

When I open the module in FGs. A white popup window comes up with the error.

[Trenloe]

When I open the module in FGs. A white popup window comes up with the error.
Thanks. I'd guessed that from br0k3n0n3's comment above.

See the XML debugging I did in post #165 - it is the db.xml file in the monsters module that has the error, not the text parse file.

[xKrisx]

So I would like to know why I don't have the full XML file for my parsed monster list? Is it possible for someone to send me the working mod?

[Trenloe]

So I would like to know why I don't have the full XML file for my parsed monster list?
It is in the module file itself. Change the .mod to .zip and unzip the resulting file - the db.xml file will be available then. Do the modification needed, zip the file back up (from within the directory, select all of the files and zip from there - do not zip it from outside of the directory), then change the .zip file to .mod. This will then be your modified module.

Or, look in the original parsed text file for the "MooncalfRogue58161" entry and remove the <x> and <aura align=top> from the Windwrack power and re-parse.

[xKrisx]

k awesome I think I have it now. Thank you

[Griogre]

I remember the mooncalf rogue having an extra tag in it. It's a very strange monster power as well, so the parser may be blowing the parse because of the tag or because of its strangeness or both. X( I've never seen a rechargeable aura as a minor action power before this monster. :p Sending you a working module would be illegal, so let me show how to fix yours since everyone who creates a module with the monster will have the same problem. This monster is a solo from Monster Vault: Threats to the Nentir Vale.

The problem is in power <id-00008> of the Mooncalf Rogue. First open the db.xml file and search on Windwrack (use a text editor or Notepad). The problems are in the <name>, <recharge>, and <keywords> elements - leave the rest the same.



<id-00008>
<name type="string"><P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P></name>
<action type="string">Minor</action>
<recharge type="string"><P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P> <P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P></recharge>
<keywords type="string"><P class="flavor alt"><aura align=top> <B>Windwrack</B> <x> Recharge 5-6</P></keywords>
.
.
.
.
</id-00008>



You need to change the red parts above to get rid of the HTML tags and fix the blown parse. I've shown the fixed lines in blue:



<id-00008>
<name type="string">Windwrack</name>
<action type="string">Minor</action>
<recharge type="string">Recharge 5-6</recharge>
<keywords type="string"></keywords>
.
.
.
.
</id-00008>

[xKrisx]

never mind. this stuff really isn't my forte.. i'm going to try one more time. i think I just missed something.

k i will try that griogre

[xKrisx]

K done and done Thank you guys very much. A+

[br0k30n3]

Or, look in the original parsed text file for the "MooncalfRogue58161" entry and remove the <x> and <aura align=top> from the Windwrack power and re-parse.

two things, it appears that maybe we should just remove all of the <x> and <aura align=top>, because it seams every time I remove them I just get another error.
Also been using notepade++ and it was crashing every time I tried to load db.xml so in case anyone else ran into this issue I started using gVim 7.4 for windows and it worked. the db.xml crashes notepad++ in addition to Dreamweaver.

I have been successful in parsing MM 1- 3 without error, so I am thinking it has something to do with one of the other the sources, this weekend when I get time I will see if I can narrow down the specific sources if possible.

[Griogre]

As mentioned above, the Mooncalf Rogue is in "Monster Vault: Threats to Nentir Vale" so if you parse any other Monster book you shouldn't have a problem. Currently the best 4E Monster books with the latest math are Monster Manual 3, Monster Vault and Monster Vault: Nentir Vale, though, if you run adventures of higher than 7th level or so.

[Griogre]

I did notice one problem with the "Add all Books to Source" button in the parser. No entry for first Adventurer's Vault is added.

[br0k30n3]

I did notice one problem with the "Add all Books to Source" button in the parser. No entry for first Adventurer's Vault is added.

I wonder if you can manually add it editing the " All Source Books.txt " .

Open All Source Books.txt and insert
<option value="10">Adventurer's Vault</option> above
<OPTION value=73>Adventurer's Vault 2</OPTION>

[Griogre]

That worked fine, thanks. I didn't know the value for AV was 10.

[Griogre]

Replying to myself is a bad habit, sorry. :p But I wanted to mention a tip to those doing parses by books who might not know. If you try to get Items by book often you will get a timeout on the server, often probably understates it - *usually* you will get a timeout. It's worth trying because it does work occasionally during non peak hours. However, if/when you do timeout there is a work around for this problem. I have always been able to get the items for a book if instead of "Select All" in the Category filter I first select Mundane Armor and add the armor, then select Mundane Weapons and add the weapons, then select Magic Alchemical Item and add the items, and so on going down the entire category list to Wonderous. It's tedious, but works everytime. Keep an eye on the new total of items to parse after you press add to make sure the current category's items were actually added.

[valeros]

New parser posted:
1) Fixes issue with Mooncalf Rogue and several other monsters, items, and traps (e.g., Gibberling Mob, Wintersnap Armor, Evard's Tentacle Trap, etc.)
2) Changed "All Source Books.txt" to try to better group related books together when displaying in the Source menu. (This also fixes the "missing" Adventurer's Vault that was previously located under Tomb of Horrors.) If someone develops a better organized listing, please post back so I can incorporate in the future.

[zapoqx]

Just wanted to mention that Mold King's Crown didn't parse cause... of a strange disease that it has (error in Compendium???), but I am not sure if it didn't parse the item or just that power.



Mold King's Crown: Not Parsed: <H1 class=atwillpower><SPAN class=level>Level 10 Disease</SPAN>Blistering Corruption</H1>
Mold King's Crown: Not Parsed: <P class=powerstat><B>Stage 0</B>: The target recovers from the disease.<BR><B>Stage 1</B>: While affected by this stage, the target takes a -2 penalty to Fortitude.<BR>
<B>Stage 2</B>: While affected by this stage, the target takes a -2 penalty to Fortitude. In addition, the target regains only half the normal number of hit points from spending a healing surge,
and the target’s hit point total cannot exceed its bloodied value.<BR><B>Stage 3</B>: While affected by this stage, the target suffers the effects of stage 2 and has an aura 2 (poison) that
cannot be deactivated. Any creature that ends its turn in the aura loses a healing surge.<BR><B>Check</B>: At the end of each extended rest, the target makes an Endurance check if it is at stage 1
or 2.<BR><I>12 or lower</I>: The stage of the disease increases by one.<BR><I>13-17</I>: No Change<BR><I>18 or higher</I>: The stage of the disease decreases by one.<BR><BR>

(p.s. Made the code with enter entries to not widen the page on accident)

[valeros]

Mold King's Crown should have parsed (the item entry part.) The disease that is part of the entry did not parse. But that disease is parsed when parsing diseases. I have a new version of the parser that removes that error. It also fixes an issue with races getting extra powers (that has always been broken as far as I can tell.) I still have to test, but hope to post soon. Thanks

[damned]

Apologies for the cross posting / thread hijacking - we're still looking for some 4E games for FG Con. And there are plenty of other games up there for any players looking for some gaming fix that weekend :)

https://www.fg-con.com/wp-content/uploads/2014/04/fg-con-newsletter.png

FG Con 4 is here!

FG Con is back for our 4th event. This time around FG Con will span 100 hours from the evening of Thursday May 8th (AUS/NZ time) to late night Sunday May 11th (US/CA Time).

We’ve got 38 sessions available for registration right now and several more close to being published. Visit the site here: https://www.fg-con.com/events and get started.

Players – now is the time to sign up for your games – but first you need to make sure you have your timezone set in your user profile – see instructions here: https://www.fg-con.com/wp-content/uploads/2014/03/FGCon_UserRegistration.pdf All players play free! Of course you still need to have installed a copy of Fantasy Grounds and TeamSpeak but you can still play even with a demo license.

GMs – there is still time to get your game listed – but don’t leave it too late as it will get harder to fill your game. Instructions for GMs here: https://www.fg-con.com/wp-content/uploads/2014/03/FGCon_GMGuidelines.pdf All Full Licenses will be bumped to Ultimate licenses for the duration of the Con so you can get those friends involved who haven’t yet committed to a license.

If you need any help – either as a player or a GM please contact the FG Con team by email: [email protected] All feedback is welcome. Suggestions regarding the site, regarding recruitment, regarding emails – whatever – the FG Con team want to hear from you.

[valeros]

Posted updated parser. Fixed issue with Races. Also updates "Power Classes.txt" for new powers.

[Geryn]

The parser keeps crashing for me if I use more than one browser.

Edit: And if I cancel the download to try and add more browsers, will I have to re-download all the items I already downloaded?

[primarch]

The parser keeps crashing for me if I use more than one browser.

Edit: And if I cancel the download to try and add more browsers, will I have to re-download all the items I already downloaded?

Hi!

I have noticed this as well.

Primarch

[valeros]

Not sure about the problem since I cannot reproduce. I re-posted the version of the parser before this update to use instead.

[aka_lump6]

I am having great difficulty with the parser. I do have an active DDI account and I have tried using both the old version and new version of the parser with the same results. Every time I have used them they crash if I select too many sections of the compendium so I limited the scope to just doing one section at a time. When I downloaded and parsed "Feats" it worked just fine all other sections gave me the following error (also see image attachment):


See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.NullReferenceException: Object reference not set to an instance of an object.
at _4eParser.FGWriters.FGCollectionWriter.firstLetter (IParsedCompendiumEntry entry)
at _4eParser.FGWriters.FGCollectionWriter.EntryInCate gory(String category, IParsedCompendiumEntry entry, CategoryField categoryField, EntryIndexTest entryIndexTest)
at _4eParser.FGWriters.FGCollectionWriter.CountEntrie sInCategory(String category, CategoryField categoryField, EntryIndexTest entryIndexTest)
at _4eParser.FGWriters.FGCollectionWriter.WriteCatego ryIndexBody(SortedSet`1 categories, String indexName, CategoryField categoryField, EntryIndexTest entryIndexTest, Boolean nameInLink, CategoryOrdering ordering)
at _4eParser.FGWriters.FGCollectionWriter.WriteAlphab eticIndex(String indexName, Boolean nameInLink, EntryIndexTest entryIndexTest)
at _4eParser.FGWriters.FGCollectionWriter.WriteIndexe s()
at _4eParser.FGWriters.FGCollectionWriter.WriteDescri ptionsAndIndexes(List`1 entries, String moduleName, StreamWriter outFile)
at _4eParser.FGWriters.FGCollectionWriter.WriteDescri ptionsAndIndexes(List`1 entries, String moduleName, StreamWriter outFile, Logger logger)
at _4eParser.FGDataFileBuilder.WriteEntryDescriptions AndIndexes()
at _4eParser.FGDataFileBuilder.Build(List`1 entries, String moduleName, String moduleCategory, String filename)
at MainViewModel.MainViewCompendiumModel.ProcessEntri es(String moduleName, String moduleCategory)
at MainViewModel.MainViewCompendiumModel.ProcessCompe ndiumEntries(String email, String password, Int32 numBrowsers, String moduleName, String moduleCategory)
at _4eParser.MainViewController.ProcessEntries()
at _4eParser.MainViewController.ProcessDownloadAndPar seEntries()
at _4eParser.MainViewController.DownloadButtonClicked (Object sender, EventArgs e)
at _4eParser.LogForm.DownloadButtonClicked(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18444 built by: FX
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
4eParser
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Users/family/Documents/4eParser_2014_0517/4eParser.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18408 built by: FX
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18408 built by: FX
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18408 built by: FX
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18408 built by: FX
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.mshtml
Assembly Version: 7.0.3300.0
Win32 Version: 7.0.3300.0
CodeBase: file:///C:/Windows/assembly/GAC/Microsoft.mshtml/7.0.3300.0__b03f5f7f11d50a3a/Microsoft.mshtml.dll
----------------------------------------
Accessibility
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18408 built by: FX
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
----------------------------------------
Microsoft.CSharp
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18408
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.CSharp/v4.0_4.0.0.0__b03f5f7f11d50a3a/Microsoft.CSharp.dll
----------------------------------------
System.Dynamic
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18408
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Dynamic/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Dynamic.dll
----------------------------------------
Anonymously Hosted DynamicMethods Assembly
Assembly Version: 0.0.0.0
Win32 Version: 4.0.30319.18444 built by: FX
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/mscorlib/v4.0_4.0.0.0__b77a5c561934e089/mscorlib.dll
----------------------------------------
System.Dynamic.DynamicAssembly
Assembly Version: 0.0.0.0
Win32 Version: 4.0.30319.18408
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Dynamic/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Dynamic.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

Any assistance you can offer will be greatly appreciated.

V/R
Lump

[valeros]

Try the following:
1) Do not use the option to download using multiple browsers (i.e., on Step 6, leave set to 1.)
2) Ensure .Net 4.0 or higher is installed.
3) Run on a Windows 7 computer (with .Net 4.0 or higher.)
4) Try parsing one of the text files that were generated from a previous run.
5) Ensure the "Power Classes.txt" file is in the same directory as the parser.

[aka_lump6]

Valeros, thanks for the response. I tried all of that and still no luck. I am running a Windows 7 system and have .NET Framework version 4.5. The strange thing is that out of all the subsections of the Compendium only Feats and Glossary have parsed properly.

[aka_lump6]

Try the following:
1) Do not use the option to download using multiple browsers (i.e., on Step 6, leave set to 1.)
2) Ensure .Net 4.0 or higher is installed.
3) Run on a Windows 7 computer (with .Net 4.0 or higher.)
4) Try parsing one of the text files that were generated from a previous run.
5) Ensure the "Power Classes.txt" file is in the same directory as the parser.

Umm I didnt see the step 6 part..tried that and it worked. As long as no extra browsers spawned it worked fine. Took forever but it did work. Thanks Valeros

[Rades]

Hi, thanks for this great tool, we just exported a nice handy DM Module and Player Module for our group and are looking forward to using the full DDI database on the board now!

I did have one question - when we exported Feats, they all came strewn together in a massive single category. It's basically impossible to use because it's 3200 feats that aren't sorted or categorized in any way, so players have to scroll through this massive list and manually browse the feat choices. Even when we separated feats by tier, the Heroic tier was still 1600 choices.

Does anyone know if there is any way to sort these? By class or prerequisite or source book or anything? I noticed the Compendium itself has the option to sort by tier (like the parser), but the Compendium also lets you sort by Source - type of book, or by specific book, while the parser only lets you choose which type of book (Rule Book, Printed Adventure, Dragon Mag, etc.)

It wouldn't be perfect, but being able to create a module that's PHB, Dragon Magazine, Martial Power 1 & 2 (for example) would let us have a Martial Feats module, which would help big-time rather than the big unsorted list.

Or if anyone has any other suggestions or solutions I'd be glad to hear it!

[damned]

my only suggestion is to use the search... just start typing and the list will start shrinking...

[Rades]

my only suggestion is to use the search... just start typing and the list will start shrinking...

I did try that, but unfortunately the Fantasy Grounds module search only searches by name...so a search of "rogue" in my Heroic Feats module brings up...1 entry, "Twilight Adept [Multiclass Rogue]", haha.

[valeros]

The lack of menu options is an Internet Explorer problem. What you need to do is:
1) Click "Select Compendium Entries" in Step 1
2) Wait for the browser at the bottom to fill-in
3) Switch to feats in the browser
4) Click the "Add All Books to Source" button (to the right of Step 2)

Now all the source books should be available so you can create modules based on books.

Also, you can combine multiple entries before downloading so you can have whatever entries you want in one file. For example, to have all the entries from all three PHB in one module you would: filter on the first PHB and select "Add Entries" in Step 2, then filter on PHB 2 and select "Add Entries" again. Repeat at will. Then do Steps 3-7 and all those entries will be combined into one module.

[Rades]

Oh fantastic, thank you valeros! This should work perfectly! :)

[Rades]

Quick question - I've parsed a list of all items together for convenience, but is there any way to make the individual categories (Magic Armor, Magic Weapons, Implements, etc) sort by level instead of alphabetically? In the old Adventurer's Vault module, all the +1 Armors are listed first, then the +2 ones, etc. I find this much easier to browse when I'm picking out appropriate treasure for my party rather than alphabetical sorting.

7164

Thanks for any help!

[valeros]

Short answer, no.

Sorting items by level would require another index (meaning writing more code.) My thought for my parser was not to be a substitute for the compendium, but just to save the time typing the entries into FG. I always assumed I would use the compendium, the module, the rulebook, etc. to find what I wanted. I just did not want to have to type it into FG once I knew what I wanted. If FG had more options for sorting built into the windows, that would make it mush easier. But right now, each sort requires an index (at least in the 4E ruleset.)

[Rades]

Okay! Thank you for the fast response, it's just a minor thing anyway. The parser modules have already made our games way easier, so thanks again for all your hard work! :D

[Shimrath]

Just wanted to pop in and tell the powers behind the throne what an excellent tool this is! A million thanks to you!

[Snoozy]

Hi guys,

I was wondering if anyone can maybe help me with this 4e parser. I started a download with it about 45 hours ago, but only selected one browser. I can't tell if the program is still downloading or not. I'm hesitant to restart my computer. It can't possible take this long, can it (1 browser)_? Any help would be appreciated. I attached a picture so you can see what I'm currently looking at.

Thanks

[Griogre]

It looks like you didn't input your DDI password? Loosely speaking it takes about 1 second per entry and some entries generate more than one writes. So on that parse probably about 10,000 seconds or about 167 minutes.

[Snoozy]

Thanks for the help. I have had issues logging into my DDI account with internet explorer. I can only get in using google chrome for some reason and when I try to log into it from the parser it give me an error saying it was unable to locate my DDI account. My account is only like 4 days old, not sure if that matters or why I have problems with this.

[Griogre]

Unfortunately, the parser uses IE internally since its a .NET program. Chrome works much better with base DDI though. I did just check and I could still log into DDI with IE.

[Snoozy]

I'll have to try on another computer then. Thanks for the help.

[huntintx]

I am having the same issue as Snoozy. I am using IE 11, it's also my only browser. Windows 7 Ulitmate, running the parser as admin. It will not let me log into DND Insider. I think this is because its pointing to the wrong websites. When I click subscribe now or or any other links on the parser I get the screen below. I have verified my email and login are correct. Ive even tried changing my password but nothing will let me sign in.

8115

Edit*

I think there's a problem with the log in. To get into DnD Insider you use a username, but the parser is requesting an email. DnD Insider is an archived link. https://archive.wizards.com/dnd/tools.aspx.

[huntintx]

I am having the same issue as Snoozy. I am using IE 11, it's also my only browser. Windows 7 Ulitmate, running the parser as admin. It will not let me log into DND Insider. I think this is because its pointing to the wrong websites. When I click subscribe now or or any other links on the parser I get the screen below. I have verified my email and login are correct. Ive even tried changing my password but nothing will let me sign in.

8117

EDIT*
I think the problem is for new accounts made in DnD Insider after they made the switch and archived the site. DnD Insider now uses usernames instead of emails. Is there anyway to change the code to reflect the changes made by DDI? Is there source code anywere?

[Griogre]

Hmm. I was able to login with the parser and make a test run but my DDI account is old. So it does seem like it might be a login rather than a path problem. I take it you tried entering your username instead of an e-mail? You might try PMing valeros but I know he doesn't have a DDI account any more.

[Dakadin]

One other thing to consider is IE 11 will update itself at times and can cause problems with some websites. Try adding the site to Compatibility View Settings for IE. Not sure if it will help but might be worth trying out.

[valeros]

Posted a new version that does not use IE for multiple downloads. Pick 2, 3, or 4 for number of browsers (higher numbers will work but will actually end up being slower.) This will generate Http Web Requests instead of using IE. Using the default on 1 will still try to use the .Net WebBrowser control, which is basically IE. No guarantees with this as I do not have a DDI account anymore.

The program does hardcode the URL, but the URL to the compendium has not changed. It is still https://www.wizards.com/dndinsider/compendium/database.aspx

[Snoozy]

Awesome! I can't wait to get home and try it. I'll let you know how it goes. Thanks for the help

[Snoozy]

I tried downloading with it today and didn't have any luck. I'm getting an error that says. No entries found. Maybe I'm doing something wrong. Thanks for the help though.

[Griogre]

WotC's server is like 3 hamsters in a cage. Assuming you actually selected something just try again at a different time.

[Redemption77]

So was the issue with the fact you login now with a username and no longer an email patched? Its refusing to work for me using one window even though Im putting in my DDI login. Since its not an email its not letting me use it.

[torndo]

im having the same problem
wont let me log in with login name or email

[torndo]

Would it be against the rules for someone to share the preparsed modules with someone who has an active d&d insider sub but it having problems getting the parser to login?

[kaynorr]

Would it be against the rules for someone to share the preparsed modules with someone who has an active d&d insider sub but it having problems getting the parser to login?

If my understanding is correct, what you're describing is unfortunately illegal. Regardless of whether or not the recipient has a DDI subscription, you're still redistributing DDI content to him/her and that's not allowed. The only way DDI content can move is from the DDI website to a subscriber, no middleman inbetween because Wizards has not granted permission for any middleman to exist.

Yes, it sucks. Intellectual Property law often does, and I (and a lot of other people) would kill for a RPG that was designed from the ground up to incorporate something like DDI only in a more open setting.

[Redemption77]

Is there something that can be changed in the coding of the parser to allow usernames instead of user emails since DDI changed their format?

[Trenloe]

...I (and a lot of other people) would kill for a RPG that was designed from the ground up to incorporate something like DDI only in a more open setting.
This describes Pathfinder (among other RPGs). All of the open material in the Pathfinder Reference Document can be included/distributed as long as the OGL is adhered to: https://paizo.com/pathfinderRPG/prd/ Hence why Smiteworks can distribute library modules with the product and why there are lots of library modules created by the community - available here: https://www.fantasygrounds.com/forums/showthread.php?16074-List-of-Modules

Or any other RPG with an OGL/SRD or creative commons licence. D&D 3.5e was the last D&D version to use the OGL (which was created by Wizards).

[Redemption77]

I need to get the modules or Parse my data from the DDI somehow other then by hand, but it doesnt seem either is feasible at this moment since one isnt ethical and the other cant be done due to program issues. Are there any other parser programs available or in the works?

[torndo]

I don't think it's the fact that they use user name rather then email addresses but more the fact that after the merger with wotc accounts new user logins aren't being added to the compendium database access. So once your logged in on the d&do insider site you have access to the compendium but can't login directly from the compendium itself unless you have an account from before the merger

[Redemption77]

Well either way its put a serious damper on people being able to use a parser for ease of scraping. Its a bloody pain in the arse doing it all by hand

[damned]

I wonder if valeros could be persuaded to go back and look at this code again... if someone were to PM him and ask him nicely...

[Redemption77]

I would love to do that, however I am while not new to Fantasy Grounds, am new to the forum and have no rapport with anyone here or a voice which means anything to anyone. It would be more like pestering if it came from me and thats not my intention in the least.

[Redemption77]

Alright everyone. I sent a message to him in his PM. Lets hope he responds. Id love to see this amazing tool working again for those of us who were late to the party

[valeros]

I posted a new parser to have the D&D Insider Tools page loaded when the program starts. From there you can login to DDI before selecting any choices in Step 1. If you stay with 1 browser, it should just navigate to the compendium and keep your DDI info. Not sure if that is what is needed or not since I do not really understand the problem and cannot test since I do not have an account anymore. Thanks

[Redemption77]

For all those who were having issues this indeed has fixed the problem. Now at the bottom of the screen where it used to show the compendium screen it now shows your login screen. You login and boom its good to go. Thank you so much Valeros

[bluedreamer]

Thank you so much Valeros
And from me too - at last I can get the Heroes of Shadows into our game
Thank you Valeros

[xKrisx]

Does this work any more?

And how do I go about getting 4E stuff again?

[bluedreamer]

It does - built new compendium up in Jan. You need to get a dndinsider account from wizards to get the data.

[xKrisx]

Okay, I'm having issues with it.

I get it running just fine and log in with my DnD insider account but a couple hours in it crashes or freezes. I try using the client that scans using internet explorer browsers but it crashes soon after starting. I've updated the .NET to the latest version. The other one it looks like it's running fine and dl'ing everything and a couple hours in stops. Not sure exactly what I'm doing wrong.

[bluedreamer]

Trying to remember what I did. I think 1st I opened IE and logged in manually to DDI. Then Add Players. Made sure it was client.xml.

Also I selected 1 thread only - and just left it.

Also I am sure it downloads a file - so you can reparse it all again without download - cannot remember how I did that or if it just left a temp file.

[xKrisx]

Hmm Well where does it download to? If I knew that maybe it wouldn't seem so frustrating..

[bluedreamer]

Mine created a txt file in the directory from which I launched it. Eg I still have "4E Player's Handbook 19218 Entries 2014_1228.txt" sitting there from a Run in Dec

[Trenloe]

The other one it looks like it's running fine and dl'ing everything and a couple hours in stops. Not sure exactly what I'm doing wrong.
Are there any messages in the scrolling window in the right side of the parser when it stops? Look for message boxes hiding behind other windows.

[xKrisx]

I have had a couple different stops on abilities. But I have managed to get a bunch of things parsed individually.
I was able to make a mod with Backgrounds, classes, deities, epic destinies, paragon paths, races, and rituals all in one. Worked
Then I went and did Feats separately. Worked
Tried doing items and I received an error out of memory. It seemed like it was actually working normaly after I created the two. Mind you I was running two parsers. One was making one file to parse everything that the PHB had, and I was running one I selected in a 2nd parse. (I'm not sure if that should be done or not but it did work for the selected entries)
I'll try again tomorrow maybe I was not doing something right.. Lol

[xKrisx]

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Runtime.InteropServices.COMException (0x80070008): Retrieving the COM class factory for component with CLSID {DC1C5A9C-E88A-4DDE-A5A1-60F82A20AEF7} failed due to the following error: 80070008 Not enough storage is available to process this command. (Exception from HRESULT: 0x80070008).
at System.Windows.Forms.OpenFileDialog.CreateVistaDia log()
at System.Windows.Forms.FileDialog.RunDialogVista(Int Ptr hWndOwner)
at System.Windows.Forms.FileDialog.RunDialog(IntPtr hWndOwner)
at System.Windows.Forms.CommonDialog.ShowDialog(IWin3 2Window owner)
at System.Windows.Forms.CommonDialog.ShowDialog()
at _4eParser.LogForm.ShowFileDialog()
at _4eParser.MainViewController.AddFile()
at _4eParser.MainViewController.AddEntriesButtonClick ed(Object sender, EventArgs e)
at _4eParser.LogForm.AddEntriesButtonClicked(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
4eParser
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Users/Kris/Downloads/4eParser_2014_1212/4eParser_2014_1212/4eParser.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.mshtml
Assembly Version: 7.0.3300.0
Win32 Version:
CodeBase: file:///C:/Windows/assembly/GAC/Microsoft.mshtml/7.0.3300.0__b03f5f7f11d50a3a/Microsoft.mshtml.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
System.Web
Assembly Version: 4.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/System.Web/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Web.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.


So I'm getting this..

[Trenloe]

It's saying "Not enough storage is available to process this command".

Don't run more than one instance of the parser.
Do you have lots of free hard drive space?
Microsoft mentions a couple of things here: https://technet.microsoft.com/en-us/library/cc978735.aspx The main one being reduce the number of programs you're running at the time and make sure you have free disk space where your paging file (for virtual memory) is running.

[xKrisx]

I thought it might have to do with that but I have 11 gigs free on my hard drive and 16 gigs ram, I don't know where I'm running out of memory from. Maybe I'm spiking and don't notice it but this was happening running a single instance of the parser as well.
Anyway I may have it working using the 0517 parser. Parsing Individual larger entries separately. I'm running powers right now. I did items and feats already. Have to see if powers finnish when I get home from class.

[LeftBreeze]

cant log in to insider
:(

[damned]

thru the parse tool or thru a web browser or because you dont have an account or forgot your password?

[LeftBreeze]

i got into insider (i had the wrong version of parser prior) but now i get the no entries found error when i try to use more than one browser 19000 files would take too long one by one

[damned]

i dont think you have much choice from previous posts...
set it and let it run.
dont select everything - do it in smaller chunks.

[LeftBreeze]

by the time you posted this i had already selected everything and now its been running for six hours too late to turn back now

[valeros]

No one has really reported trouble using the single browser. So it is slow, but you can download in chunks (like monsters, then feats, then powers, etc.) Once you have the downloaded files, you can combine them to create a module that has all the entries you want in it. That is what the "Select File" option is for. You can load as many of the downloaded .txt files (not the .mod files) and it will create a new txt and mod file by combining those files.

[kevininrussia]

I was able to scrape DDI and load into Fantasy Grounds. I noticed the powers don't have the flavor text. Is this normal operation or did I do something wrong? Is there a way to add the flavor text without having to do it manually?

[Griogre]

That is expected behavior from the parser. I don't remember if DDI has flavor text in the entries or not - though I don't think it did originally, but I don't have a subscription any more.

Edit: Welcome to the boards, by the way. :)

[kevininrussia]

That is expected behavior from the parser. I don't remember if DDI has flavor text in the entries or not - though I don't think it did originally, but I don't have a subscription any more.

Edit: Welcome to the boards, by the way. :)

Thanks for the reply and welcome :-)

I checked DDI. The flavor text is there so I will copy the text for the players chosen powers.

The Item Parser worked great when using only one browser instance. When adding two or more it would fail. My OS is Windows 8.1. The "Players Handbook" option took 7 hours to download. I mention it just as reference for others who might try in the future. Overall its quite an impressive tool!

[Trenloe]

I was able to scrape DDI and load into Fantasy Grounds. I noticed the powers don't have the flavor text. Is this normal operation or did I do something wrong? Is there a way to add the flavor text without having to do it manually?
The flavour text is available in the DDI Compendium and is present in the modules produced by this parser. However, the module stores the data in a field <stringflavor> while the 4E ruleset looks for a field <flavor> so it never finds the flavor text.

The quick way to fix this is to open client.xml in the module file and replace stringflavor with flavor for all entries and re-save the module file. This will show the flavour text, for example:

https://dl.dropboxusercontent.com/u/39085830/Screenshots/Fantasy%20Grounds/4E%20Acrobat%20Trick%20Power.jpg

[kevininrussia]

The flavour text is available in the DDI Compendium and is present in the modules produced by this parser. However, the module stores the data in a field <stringflavor> while the 4E ruleset looks for a field <flavor> so it never finds the flavor text.

The quick way to fix this is to open client.xml in the module file and replace stringflavor with flavor for all entries and re-save the module file.

That fixed it. Thanks!