4E Item Parser

That is the way the Compendium appears in Internet Explorer. The application is using IE for that window. So if you launch IE and go to the Compendium, that is what the menu looks like. So I do not believe it is an issue with this application, but with how IE renders the page. Nothing I can do about that.

Chrome seems to render the menu correctly. And I have not tried this, so have absolutely no idea if it will actually work. But if you load the Chrome overlay on top of IE, maybe then IE (and thus this application) would work???

Yeah in just the compendium it sorts right with chrome. I normally use chrome so I didn't know about IE that's why I thought it was weird. :p When I get a chance I'll research the overlay more and see what happens.

Hi!

Valeros, with 5e playtesting wrapping up there is a lot of talk about the "demise" of DDI.

Although I have already future proofed myself with your parser, was there any data left in the compendium your parser couldn't get?

Thanks!

Primarch

As far as I know, it should work with everything but "Companions & Familiars". I never did those entries.

Quote:

---

Originally Posted by valeros

As far as I know, it should work with everything but "Companions & Familiars". I never did those entries.

---

Hi!

Are you planning to include those in the parser, or is the parser "done".

Thanks!

Primarch

I do not really have the desire to do those entries, so basically "done". But willing to try to fix bugs or potentially modify if a new version of FG or the ruleset needs output in a different format.

Quote:

---

Originally Posted by valeros

I do not really have the desire to do those entries, so basically "done". But willing to try to fix bugs or potentially modify if a new version of FG or the ruleset needs output in a different format.

---

Hi!

No problem. I just wanted to know since I have multiple backed up copies of the entire compendium and the raw data to manipulate I don't really need a DDI subscription anymore.

Thanks!

Primarch

I just parsed all the items and I thought I'd mention there is one item that throws an error, on the Mold King's Crown.

So... I finally used the parser for the first time. How incredibly amazing this thing is.

So... if I want to strip out anything NOT in the 4E Essentials books, can I just go through the db.xml file and delete each node I don't want? (I understand there's a LOT of them.)

Yeah, that would work. Technically you really only have to update the indexes but that's most of the work anyway since there are multiple index entries per data point. Its unfortunate the Select Compendium Entries doesn't work right. It's a bug in their code for Internet Explorer where some entries don't appear after a dash. I've complained to WotC, and they might fix it if enough people complained - just because I assume they are going to use the same DDI framework for 5E. I'm not sure if you noticed it but the Other tab has page numbers and source(s) so if you trust you players not to mess up you could just tell them to check.

Did you ever figure out an IE overlay addon to get the Source filter to populate correctly? I wish it was possible to force the Parser to use Chrome. :(

All finished! Wasn't too bad. Items, Powers, and Creatures could all be filtered to just scrape the Essentials stuff, so the only really big list I had to edit was Feats. Collapsed all the nodes and just went to town clearing them out, then made the index match. After that I took my Monster Vault tokens and scanned them (never punched them out of the cardstock) then made a set of tokens with TokenTool. Finally I edited the token entries for all 305 creatures (since so many of them use the same token). I broke the DM stuff and MM stuff into two different modules though.

Anyway, all finished building my Essentials modules! Thanks again for the amazing utility. I've done a full 4E PHB scrape, 4E DMG scrape, and then I also did individual scrapes for each category as well, just in case the 4E DDI dies sooner rather than later.

Hi All,

First, a big thank you to Valeros and everyone who has been involved with the parser; it's amazingly useful. I've been having trouble getting the Powers to parse. I tried using the player entries, as well as trying to scrape and parse the powers separately. The scraping seems to go well, but all the powers fail to parse. All of the other entries seemed to work without issue. Has anyone else encountered this issue, and if so, can you share how you fixed it? Thanks!

I ran the parser last night to check after seeing monkinmeditiation's post and there are a ton of fail to parse entries. I presume 1 for each power. One thing I noticed when I ran it last night, that when gathering powers the number found were a lot more than when I ran it last: i.e. 7 thousand plus vs. 4,600ish (I only parse rulebook powers) and obviously there hasn't been any new rulebooks published since I last ran that module in August.

Yeah, when I scraped all of the powers, I got something like 9,380 entries, and every single one of them failed to parse. All of the other categories (Player Entries and DM Entries) seemed to work just fine otherwise and I was able to load the modules without incident. I have two copies of the powers .txt files (if that matters).

Thanks, Griogre for checking on this.

Sorry powers stopped working. Unfortunately, I have already let my DDI subscription lapse since there did not seem to be any updates. So there is not a good way for me to troubleshoot the problem. The parser does rely heavily on the HTML, so if that changed, it could cause a problem. Also, the parser was written for Windows 7. If you are running Windows 8 and a newer Internet Explorer, it is unlikely but possible that is the cause.

Hi Valeros. Thanks for replying. I am using Windows 7, for what it's worth. Being respectful of your time, do you have any suggestions for what I might be able to do? My DDI subscription lasts for another 2 weeks or so.

Thanks again.

If you email me the powers txt file (or 3+ powers copied out of the file), I will see if I can quickly fix the issue. Since all powers are failing, hopefully there is a single easy fix.

I'm running Win7 as well. Hopefully its an easy fix.

Hi Valeros,

Thank you very much for your kind offer to help. I'll send you the file as soon as I get back to my computer.

Quote:

---

Originally Posted by valeros

If you email me the powers txt file (or 3+ powers copied out of the file), I will see if I can quickly fix the issue. Since all powers are failing, hopefully there is a single easy fix.

---

Thanks again to Valeros; he kindly took the time to look at my file. It turns out it was user error :o; I had moved the parser from its original folder and left the Power Classes.txt file in the folder. Once I moved the scraped text file into the same folder as the parser and Power Classes.txt file, it worked perfectly. Sorry for the false alarm.

Yeah, and I was an idiot. I found I had deleted the power file while I was cleaning up the modules in the folder. :p

With the DDI for 4E's future becoming murky I started preping to do the final parses after the last Dungeon Magazine for 4E is published in December. I noticed a problem with monsters not having the leader keyword, ie a Frost Goblin Hexer which is a "Level 3 Elite Controller (Leader)" npc entry only lists him as a "Level 3 Elite Controller." Anyone else have this problem?

Looks like a bug. I attached a new version on the first post. I left the old one just in case I introduced some other bugs because between the last release and this fix I had changed a lot of code doing code cleanup.

I got a chance to try the new parser and got an error about using an abstract constructor. I've attached a screen shot so you can see the selections but it was just an all items for the DM test run:

https://i105.photobucket.com/albums/...psdeb4675a.png

Edit: After glancing at the stack trace I decided to try running the parser with only one browser and without entering the e-mail and password for the my DDI account in the Parser Interface. The parser then ran, but one browser is pretty slow. :p

Edit 2: This did complete with the following item errors. I remember the Mold King's Crown didn't work last time but the others are new:

Wintersnap Armor: Not Parsed: [<H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura) Daily (Minor Action)</H2>] in ⫪㐽魏뉛袏䍺੐
Wintersnap Armor: Not Parsed: <H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura) Daily (Minor Action)</H2>
Brassburn Armor: Not Parsed: [<H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Attack Power (Aura, Fire) Daily (Immediate Reaction)</H2>] in ⫪㐽魏뉛袏䍺੐
Brassburn Armor: Not Parsed: <H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Attack Power (Aura, Fire) Daily (Immediate Reaction)</H2>
Armor of Scintillating Colors: Not Parsed: [<H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura, Charm) Daily (Minor Action)</H2>] in ⫪㐽魏뉛袏䍺੐
Armor of Scintillating Colors: Not Parsed: <H2 class=mihead><IMG src="https://www.wizards.com/dnd/images/symbol/aura.png" align=top>Utility Power (Aura, Charm) Daily (Minor Action)</H2>
Mold King's Crown: Not Parsed: <H1 class=atwillpower><SPAN class=level>Level 10 Disease</SPAN>Blistering Corruption</H1>
Mold King's Crown: Not Parsed: <P class=powerstat><B>Stage 0</B>: The target recovers from the disease.<BR><B>Stage 1</B>: While affected by this stage, the target takes a -2 penalty to Fortitude.<BR><B>Stage 2</B>: While affected by this stage, the target takes a -2 penalty to Fortitude. In addition, the target regains only half the normal number of hit points from spending a healing surge, and the target’s hit point total cannot exceed its bloodied value.<BR><B>Stage 3</B>: While affected by this stage, the target suffers the effects of stage 2 and has an aura 2 (poison) that cannot be deactivated. Any creature that ends its turn in the aura loses a healing surge.<BR><B>Check</B>: At the end of each extended rest, the target makes an Endurance check if it is at stage 1 or 2.<BR><I>12 or lower</I>: The stage of the disease increases by one.<BR><I>13-17</I>: No Change<BR><I>18 or higher</I>: The stage of the disease decreases by one.<BR><BR>
3728 Total Entries
10495 Total Resulting Parsed Entries
done

I have a request for the parser, could you please set all items in modules to identified? Currently all the items are marked as unidentified (or more correctly because there isn't an identification element - the ruleset defaults to unidentified). This means players with lite versions of FG can never see the magic item descriptions in modules and while they can see normal equipment as soon as they drag and drop it into their character's inventory it becomes an unidentified object.

This is happening because I use the "Item: Identification On" setting. Personally, I can care less about identified/unidentified status and I would normally just set it off - but when you do so it shuts off the note field on the item, which I do find very useful.

The element for setting identification is below with the 1 meaning true.

Code:

---

<isidentified type="number">1</isidentified>

---

While this can be fixed with a selected lines mass find an replace, I'm probably not the only DM who has player's complaining. :(

Quote:

---

Originally Posted by Griogre

While this can be fixed with a selected lines mass find an replace, I'm probably not the only DM who has player's complaining. :(

---

Another option would be to add another game option to 4E that forces all items as identified by default. This should be achievable by overriding CoreRPG/4E's default item window class so that it always sets isidentified to 1. That way you will always have access to the notes. Or change the 4E item campaign/reference window classes so that notes are always displayed. I can add it to the to do pile if you think it would be useful option moving forward.

Dr. Zeus there is an option that forces all items to be identified by default. Unfortunately it caused the note section on the magic item window to not display. Personally, I think it would be better to make the note field always display. I've found it useful to note which character's wishlist the item is from and the encounter it was found in. I've also found the reference book and page number the parser puts in note field for each item to be nice bonus.

If notes always on was slipped into the 4E item campaign/reference window classes in the future, that would be great.

Added new version (2013_1116) 1) Added all items in modules are identified. 2) Mold King's Crown - only the disease entry added to the Crown entry is not parsed with the Crown entry. The Crown item itself is parsed. The disease can be parsed under diseases. 3) Sorry, for the crash, not sure if I fixed the root issue. I was able to reproduce the crash, and this new version does not crash for me so hopefully it is fixed. Also, the new version should give a little bit better info on a crash if it crashes. 4) Sorry, not sure on the armor items either. I am not getting those errors with either the old or new version so I cannot reproduce the error. Maybe those three items have changes since the last time I downloaded entries. And I no longer have my DDI subscription to check the discrepancy.

I have seen some video and some screen shots where people have been able to add individual modules for each rule book. I was wondering is there was a tutorial around that would demonstrate how to parse from a specific source. I played around with DDI Compendium was looking to extract all classes from that source the Compendium only shows two classes.

EDIT:

Ok so I see what may be my issue , so I click on select Compendium Entries. Now on the integrated browser it has the Compendium , for some reason under source the drop down list does not show anything beyond the -


Update:
The problem with the drop down menu appears to be a issue with Internet Explorer and CSS

I put a new version (4eParser_2014_0201.zip) on the first post. There is now a button to the right of "Add Entries" called "Add All Books to Source". If you have already selected "Select Compendium Entries" and have waited for the page to finish loading, you can then use this button to add all the Source Book options to the "Source" filter. This adds all books, not just the relevant ones. Also, if you use this, you must have the "All Source Books.txt" file in the same directory as the parser (just like it requires the "Power Classes.txt" for powers.) You only have to use this button once, not each time you go to a new tab.

Thanks Valeros, I'll try this.

Quote:

---

Originally Posted by valeros

I put a new version (4eParser_2014_0201.zip) on the first post. There is now a button to the right of "Add Entries" called "Add All Books to Source". If you have already selected "Select Compendium Entries" and have waited for the page to finish loading, you can then use this button to add all the Source Book options to the "Source" filter. This adds all books, not just the relevant ones. Also, if you use this, you must have the "All Source Books.txt" file in the same directory as the parser (just like it requires the "Power Classes.txt" for powers.) You only have to use this button once, not each time you go to a new tab.

---

This is awesome , Thanks Sir

Hi i have used the new updated parser on the players hand book with nps at all, when it came to parsing the dungeon masters stuff i got this error when i loaded up fantasy grounds so it wont load in the modules section

Database Error: A XML parse error occurred processing file 4E Dungeon Master's Guide:db.xml - Error on line 306486: Error reading end tag.

if someone could please help me fix this it would be awesome thank you for your time :)
Spike

Hey! There seems to be a bug with the new version, that causes a whole bunch of source book filters to appear, and none of them actually work as far as I can tell. Happens if you search for whatever, click something, so you open up a popup with login confirmation, and then detailed information about whatever you just clicked on. When you close that the filter is messed up.

It doesn't really bother me, since I can just open the compendium on a regular browser at the same time. And thanks, really, for an awesome parser!

This is a bit off topic, but as far as I know varelos has no plans to incorporate familiar, companions or summons parsing to his parser, and the other parser doesn't support companions or summons either. Is there a parser that does?

Quote:

---

Originally Posted by spike1288

Hi i have used the new updated parser on the players hand book with nps at all, when it came to parsing the dungeon masters stuff i got this error when i loaded up fantasy grounds so it wont load in the modules section

Database Error: A XML parse error occurred processing file 4E Dungeon Master's Guide:db.xml - Error on line 306486: Error reading end tag.

if someone could please help me fix this it would be awesome thank you for your time :)
Spike

---

I got the same issue, I have even attempted to package the Module manually , getting the same result

So I do not have my DDI subscription anymore, so hard for me to troubleshoot. To my knowledge, the only change to the most recent parser was for the "Source" option. Nothing else should have changed. And to my knowledge, no data in the Compendium should be changing since I do not think WoTC is updating that anymore. If you parse the downloaded data file (i.e., do not download the data again) with the oldest parser still posted, does the error still happen? If so, that probably means something changed in the data or the data downloaded incorrectly (although that does not seem likely since multiple people are having the issue.) If not, then it is the parser. But if that is the case, you should now have a working module. Another option is to try one of the older parsers and download all the data again. But if the previous step did not work, this is not likely to work either.

Quote:

---

Originally Posted by valeros

So I do not have my DDI subscription anymore, so hard for me to troubleshoot. To my knowledge, the only change to the most recent parser was for the "Source" option. Nothing else should have changed. And to my knowledge, no data in the Compendium should be changing since I do not think WoTC is updating that anymore. If you parse the downloaded data file (i.e., do not download the data again) with the oldest parser still posted, does the error still happen? If so, that probably means something changed in the data or the data downloaded incorrectly (although that does not seem likely since multiple people are having the issue.) If not, then it is the parser. But if that is the case, you should now have a working module. Another option is to try one of the older parsers and download all the data again. But if the previous step did not work, this is not likely to work either.

---

I looked at the XML file that it made, I didn't see anything that looked off, Possibly because to the size or number of entries ? I'll try to use it and do it for each course separately. I was having issues prior to the new update with the DM entries, just didn't dive into it with any depth until now.

So every time I try and scrape the creatures info using the Parser I get a 'Database Error: A XML parse error occurred processing file Monsters:db.xml - Error on line 239812: Error reading tag.'

What is going on and how is that fixed? I can parse everything in the DM compendium except the Creature entries. Any help would be awesome.

Quote:

---

Originally Posted by xKrisx

So every time I try and scrape the creatures info using the Parser I get a 'Database Error: A XML parse error occurred processing file Monsters:db.xml - Error on line 239812: Error reading tag.'

What is going on and how is that fixed? I can parse everything in the DM compendium except the Creature entries. Any help would be awesome.

---

There is an XML format error in the monsters db.xml file - this will be caused by some issue in the data downloaded from the DDI website. It can sometimes be possible to identify what the error is and manually modify the db.xml file and then run this through the parser again (don't re-scrape from the website).

Open the file in an XML enabled editor (Notepad++ for example - which is free) or a XML validator website such as this: https://www.w3schools.com/xml/xml_validator.asp Although the website may not be able to process such a large file. These may help you to identify where the XML error is and fix the issue. The problem is that sometimes the line where the error is reported is not where the error occurs...

If that doesn't work for you, try posting some of the data here - perhaps 20 or so lines either side of line 239812 to see if it is something obvious. If you're not sure where that line is, open the file in Notepad++ (https://notepad-plus-plus.org/) which will show you the line numbers.