Tenian's Parser - Version 4.1.1.1+ Beta

[EugeneZ]

Hi all!

If you're in a rush, here's the skinny:


If you don't have the latest stable version (https://tenian.eugenez.net/Parser/4EParser_4_0_118.msi) of the Parser installed, install it first (https://tenian.eugenez.net/Parser/4EParser_4_0_118.msi)!
Download the .NET 4.0 Redistributable (https://msdn.microsoft.com/en-us/netframework/aa569263) and install it.
Download the beta release of the Parser (https://www.eugenez.net/downloads/parser/4.1.2.3/Parse4E.exe) and copy it to the application directory where you installed the stable Parser, overwriting the previous Parse4E file. Most likely, this is in "C:\Program Files\4EParser"


Release Notes
If you're not in a rush, here's extra info!

Tenian developed an application that allows you to parse DDI materials. He did this for himself but kindly released it to the public. However, he is currently on a DnD hiatus and has ceased development of the application. He does not want the application's source code to be released since it is his. But he has given me the source so that I may maintain it for bugs and feature requests, as time allows.

There are a few problems with this:


I don't use the same development tools Tenian does. His are 1) old and 2) expensive (Visual Basic/Studio 2003) . Mine are 1) new and 2) free. (Visual Basic Express 2010) I don't have the funds to correct this, leading to the following bullet points:
Visual Studio Express 2010 requires that end-users install the 4.0 version of the .NET framework (https://msdn.microsoft.com/en-us/netframework/aa569263). I could have sworn there was an option somewhere to change that, but whether its because it's Visual Basic (I'm much more familiar with C#) or because it's an imported project or my memory's just bad, I see no option to change the dependency to v2.0, the version Tenian developed on.
I cannot use the installer than Tenian developed since it is specific to his software. I plan to eventually use Nullsoft's free, open source installer platform, but setting this up to match Tenian's setup is a daunting task. Until I get around to this, my releases will be beta releases only, with no installer, just a binary (.exe) file you will paste over the Parse4E.exe file from Tenian's latest stable 4.0.118 version (https://tenian.eugenez.net/Parser/4EParser_4_0_118.msi). Sorry for the inconvenience, but its better than nothing! Finally, the changelog will be posted here on the forums until I get the installer up and running as well.
Tenian's software is well over 10,000 lines of code (maybe 20k or 30k, I just scanned the files in about 10 seconds). He coded for himself, not for others. He understood his own code, and I do not. I try to understand the components I modify to the best of my ability. I also lack the experience Tenian has with knowing what type of errors to watch out for. I try to test the software I release, but to be frank, I'm confident that you will not receive the level of quality you expected from Tenian. Recall all the bugs in the early days of the software. I will fix bugs as quickly as they are found, but for that to happen, they must be reported. Preferably, in this thread. My apologies in advance for the bugs!


Download
The latest beta release can be found here:

https://www.eugenez.net/downloads/parser/4.1.2.3/Parse4E.exe

Please report any bugs you find.

Requirements

.NET Framework 4.0 (https://msdn.microsoft.com/en-us/netframework/aa569263)
An installed copy of version 118. (https://tenian.eugenez.net/Parser/4EParser_4_0_118.msi) (Copy the above binary over the old binary.)


Documentation
Some people have developed some wiki documentation (https://oberoten.dyndns.org/fgwiki/index.php/4EParser) for the parser here (https://oberoten.dyndns.org/fgwiki/index.php/4EParser).

Changelog
v.4.1.2.3

DrZeuss identified and fixed the item filtering issue. Items should work again.

v.4.1.2.1

Errant string <p class="publishedIn"> removed from scrape

v.4.1.1.1

Fixed Item scraping from the Compendium
Fixed Item HTML output
Lowered inter-item pause significantly, scrapes should now be much faster

[Mooses8D]

Wanna be the first to thank you for giving a great effort to seeing this application work. I have a scrape going right now. If there is any feedback we can give you that would help, please let us know, and again thank you for your time and effort.

---===Edit===---

Ok, finished up that Scrape and attempted to Parse it. I got a few errors that I could correct using Notepad++, like a few missing </p> tags and such.

The main issue I'm running into now is when I try to parse everything I got from the Player's Handbook is when it gets to Powers. I run into something that looks like this

7/11/2011 2:45:01 AM : ERROR:System.Xml.XmlException: '<' is an unexpected token. The expected token is '>'. Line 1521, position 342.

<keywords type="string">Arcane, Force, Implement</keywords>
<action type="string">Standard Action</action>
<range type="string">Ranged 20</range>
<source type="string">Wizard Attack 1</source>
<description type="formattedtext"><table><tr><td><b>Target:</b>One creature or object</td></tr></table><table><tr><td><b>Attack:</b>Intelligence vs. Reflex</td></tr></table><table><tr><td><b>Hit:</b>2d8 + Intelligence modifier force damage. Make a secondary attack.</td></tr></table><table><tr><td><b>Secondary Target:</b>Each enemy adjacent to the primary target</td></tr></table><table><tr><td><b>Secondary Attack:</b>Intelligence vs. Reflex</td></tr></table><table><tr><td><b>Hit:</b>1d10 + Intelligence modifier force damage. <p class="publishedIn"></p></td></tr></table></description>


*** <shortdescription type="string">Target: One creature or object; Attack: Intelligence vs. Reflex; Hit: 2d8 + Intelligence modifier force damage. Make a secondary attack.; Secondary Target: Each enemy adjacent to the primary target; Attack: Intelligence vs. Reflex; Hit: 1d10 + Intelligence modifier force damage. <p class="publishedIn"></p</shortdescription>
<class type="string">Wizard</class>
<powertype type="string">Attack</powertype>
<level type="number">1</level>
<tier type="string">Heroic</tier>
<type type="string">Power</type>


The offending text is in Bold and Underlined. I can't seem to edit that particular tag because it doesn't show up in Notepad++, or I don't know a way to make it show up. Currently using Win7(x64bit), and using the 4.1.1.11 version of 4e Parser. If there is a way to configure Notepad++ to show the text the Parser is referring to I think I can fix this up.

Thanks again for your time and effort, and the help anyone else provides.


---===Edit #2===---

I was running into this same error using 4e Parser 4.0.118.

[ksm]

Thanks EugeneZ!

I am parsing the PHB right now, adn will let you know how it goes (so far so good).

[arotter]

@EugeneZ: thanks for the update! :D Alas, I still have the same problem (see attached screenshot). No scraping for me. :(

I've reinstalled the parser and applied your binary to it and also tried running it explicitly as Administrator. No success.

If you want I can try to trace the sent data during a scrape attempt with a network sniffer. Thinking of it, I'll probably do it anyway since I'm messing around with the Compendium API myself currently (trying to get an AS3 lib out of it for an AIR application).

And last but not least: don't apologize for any bugs. You're doing a great job here. I know myself how difficult it is to work with code that you haven't written yourself (and that probably wasn't written with readabiliy in mind). :)

[Trenloe]

Ok, finished up that Scrape and attempted to Parse it. I got a few errors that I could correct using Notepad++, like a few missing </p> tags and such.

The main issue I'm running into now is when I try to parse everything I got from the Player's Handbook is when it gets to Powers. I run into something that looks like this

7/11/2011 2:45:01 AM : ERROR:System.Xml.XmlException: '<' is an unexpected token. The expected token is '>'. Line 1521, position 342.

<keywords type="string">Arcane, Force, Implement</keywords>
<action type="string">Standard Action</action>
<range type="string">Ranged 20</range>
<source type="string">Wizard Attack 1</source>
<description type="formattedtext"><table><tr><td><b>Target:</b>One creature or object</td></tr></table><table><tr><td><b>Attack:</b>Intelligence vs. Reflex</td></tr></table><table><tr><td><b>Hit:</b>2d8 + Intelligence modifier force damage. Make a secondary attack.</td></tr></table><table><tr><td><b>Secondary Target:</b>Each enemy adjacent to the primary target</td></tr></table><table><tr><td><b>Secondary Attack:</b>Intelligence vs. Reflex</td></tr></table><table><tr><td><b>Hit:</b>1d10 + Intelligence modifier force damage. <p class="publishedIn"></p></td></tr></table></description>


*** <shortdescription type="string">Target: One creature or object; Attack: Intelligence vs. Reflex; Hit: 2d8 + Intelligence modifier force damage. Make a secondary attack.; Secondary Target: Each enemy adjacent to the primary target; Attack: Intelligence vs. Reflex; Hit: 1d10 + Intelligence modifier force damage. <p class="publishedIn"></p</shortdescription>
<class type="string">Wizard</class>
<powertype type="string">Attack</powertype>
<level type="number">1</level>
<tier type="string">Heroic</tier>
<type type="string">Power</type>


The offending text is in Bold and Underlined. I can't seem to edit that particular tag because it doesn't show up in Notepad++, or I don't know a way to make it show up. Currently using Win7(x64bit), and using the 4.1.1.11 version of 4e Parser. If there is a way to configure Notepad++ to show the text the Parser is referring to I think I can fix this up.

What do you have in powers.txt from the scrape for "Force Orb Wizard Attack 1"?

When you fixed up the missing </p> tags did you add new </p> tabs in or remove the offending opening tag - usually <p class="publishedIn"> ?

I usually remove the offending tag.

The error you're seeing looks like you've been adding the close tag </p> and it somehow got messed up here by only adding </p

[Mooses8D]

@EugeneZ: thanks for the update! :D Alas, I still have the same problem (see attached screenshot). No scraping for me. :(

I've reinstalled the parser and applied your binary to it and also tried running it explicitly as Administrator. No success.

If you want I can try to trace the sent data during a scrape attempt with a network sniffer. Thinking of it, I'll probably do it anyway since I'm messing around with the Compendium API myself currently (trying to get an AS3 lib out of it for an AIR application).

And last but not least: don't apologize for any bugs. You're doing a great job here. I know myself how difficult it is to work with code that you haven't written yourself (and that probably wasn't written with readabiliy in mind). :)

@arotter
Can you have plus signs (+) in an email address? Are you entering it in correctly? I used the Compendium just now to scrape the PHB, and I finally got it working. Maybe check the email address field again.


---===Edit===---

Also, maybe log into your WoTC D&D Insider using a browser and then try running the scrape? I have it set to remember me being logged in, and that seems to also be a working combination.

---===Edit #2===---

Also, that Account Validation error popped up once or twice while I was using the Scrape but it was during heavy internet traffic so the connection might have dropped out. I found that the information it had grabbed was indeed there, just not complete. Maybe try running it during non-peak hours.

[Mooses8D]

What do you have in powers.txt from the scrape for "Force Orb Wizard Attack 1"?

When you fixed up the missing </p> tags did you add new </p> tabs in or remove the offending opening tag - usually <p class="publishedIn"> ?

I usually remove the offending tag.

The error you're seeing looks like you've been adding the close tag </p> and it somehow got messed up here by only adding </p

@Trenloe
Was just about to post that I had found a work around.

But to answer your question, yeah I was closing all the <p class="publishedIn"> lines by adding </p> to close them up, but what I ended up doing to fix it was adding an additional > (</p>>) at the end so that it would continue onto the <shortdescription> line.

It appears to have worked and I now have the Player's Handbook in my library. Thank you for the quick response.

[arotter]

@arotter
Can you have plus signs (+) in an email address? Are you entering it in correctly? I used the Compendium just now to scrape the PHB, and I finally got it working. Maybe check the email address field again.
Yes, you can. A description about it can even be found on Microsoft's Hotmail homepage (https://explore.live.com/windows-live-hotmail-plus-addresses-using?mkt=en-US). Many services I use accept such email addresses (even WotC's homepage works perfectly with it).

I've copied and pasted the email address from a text editor both into the Compendium's login screen and into the parser (and did the same for my password). The result was that the Compendium logged me in while the parser told me that my account couldn't be validated. :confused:


---===Edit===---

Also, maybe log into your WoTC D&D Insider using a browser and then try running the scrape? I have it set to remember me being logged in, and that seems to also be a working combination.
Hmmm... I've just tried it and got the same results. The parser asks for my password and I get the typical "Compendium returned a login screen..." message at each "Processing <...>" entry.


---===Edit #2===---

Also, that Account Validation error popped up once or twice while I was using the Scrape but it was during heavy internet traffic so the connection might have dropped out. I found that the information it had grabbed was indeed there, just not complete. Maybe try running it during non-peak hours.
Well, on my end I have a 100MBit cable connection which was always idle during my attempts. I've also been trying it several times over the last weekend at very different times so I somehow cannot imagine that this is really my problems' source.

But thanks for the suggestions, thou. Any help or suggestion is appreciated. :)

[Mooses8D]

@arotter

Maybe it's a Firewall issue?

[ksm]

Scraping the PHB worked. However when I try to parse the results I get an error;


11/07/2011 11:45:40 PM : ERROR:System.Xml.XmlException: The 'p' start tag on line 586 position 212 does not match the end tag of 'description'. Line 587, position 7.

<cost type="number">25</cost>
<type type="string">Light</type>
<prof type="string">Leather</prof>
<description type="formattedtext">
<p>Leather armor is sturdier than cloth armor. It protects vital areas with multiple layers of boiled-leather plates, while covering the limbs with supple leather that provides a small amount of protection.</p><p><p class="publishedIn"></p>
*** </description>
</leatherarmor>
<hidearmor>
<name type="string">Hide Armor</name>
<ac type="number">3</ac>
<min_enhance type="number">0</min_enhance>


I think I can see were the extra <p> tag is, but it looks like I might need to edit the files to fix - and it may have done this in several locations?

So is this the Compendium spitting back malformed data, or something in the parser?

[Mooses8D]

Scraping the PHB worked. However when I try to parse the results I get an error;



I think I can see were the extra <p> tag is, but it looks like I might need to edit the files to fix - and it may have done this in several locations?

So is this the Compendium spitting back malformed data, or something in the parser?

@ksm
Using Notepad++ you can search for the offending tags specifically, and change them all at the same time using the Replace function. The file you need to edit is in the input folder (assuming you followed the tutorial on the Fantasy Grounds Wiki about Scraping) where you selected your scraped files to be dropped into. It should be a text document titled powers.txt or rituals.txt and such.

[ksm]

Yeah, I've done that in some files from the previous version of the parser.

Now at least the scrape is 100% successful.

Many thanks to EugeneZ for putting in all this work.

[ShaneB]

All those that are getting tag errors... its a fairly simple fix actually.. just delete the offending tag in the .txt file (for Monsters Vault this works, I would hazard a guess it will work for everything.)

Go into the file in notepad

Edit->Replace

replace <p class="publishedIn"> with nothing.

With Monster Vault you will also need to replace <h1 class="atwillpower"> with nothing

[bnason]

New FG2 user, and I have to say - this tool is freaking awesome. TY for updating it, no idea what the prior versions were like but yours has been flawless, so far.

Couple of questions:

[/list]
Is there a way to scrape more than one source book at once? I get that I can get all the PHBs, for example, by doing one at a time and appending to the same directory, but ...!
What is the "Backgrounds" option scraping when I select it?
How do I attach tokens/images automatically? I can see the menu options in parse mode, but not sure how to setup the token image files correctly to make this work.
[/LIST]

[Zeus]

@Eugenez: 4.1.1.1 installs fine for me. Scraping several books now, will let you know how I get on.

@bnason:


Is there a way to scrape more than one source book at once? I get that I can get all the PHBs, for example, by doing one at a time and appending to the same directory, but ...!

Not that I am aware of. As you have said you can scrape each one sequentially and append the data but thats it.


What is the "Backgrounds" option scraping when I select it?

Backgrounds refers to Regional Background and Traits data available in the Compendium for some campaign worlds. Basically characters from certain regions gain benefits. See the Forgotten Realms Players Guide for an example set of regional benefits for the continent of Faer&#251;n.


How do I attach tokens/images automatically? I can see the menu options in parse mode, but not sure how to setup the token image files correctly to make this work.

For NPCs, place your tokens in a folder and point the parser's npc entry tokens folder to it. Then match the name of the tokens (in lowercase) to the names of the corresponding NPC entry name in your npcs.txt file. The parser will then automatically attach the token to the NPC data in the module you are building.

Otherwise you have to manually link the tokens for each NPC/group of NPCs, using the ZToken notation in the npcdata.txt file.

Linking images for Story and Reference text is achieved using the ZLINK notation.

[Zeus]

@EugeneZ:

Scraping seems to be working just fine, I scraped severed books worth of content with no runtime errors.

Parsing the scraped content on the other hand generally failed due to either equipment, powers or familiars content containing an offending and unclosed <p class="publishedIn"> tag as part of the item, power or familiar description. Removing the offending tag from all files using search and replace fixes the problem. The files appear to parse OK after that.

[EugeneZ]

Thanks for the reports and testing, everyone. I just don't have the time to properly test this thing so these updates are only possible thanks to you guys. New release coming this weekend!

@arotter: Can you contact me on IM? It'll be easier to diagnose what the problem is so that we can solve it once and for all. My AIM screenname is eugene0506, my MSN/gChat/e-mail is [email protected], and my skype is "eugene.zar".

@DrZeruss: Looks like that tag you bolded is indeed the last remaining major issue... everyone else in this thread seems to be facing the same issue. I'll take care of it this weekend, thanks!

If there's any feature requests, this thread would be a fine place to put them. I probably won't be able to get to them for a few weeks (I have a vacation coming up) but there's no reason not to start a list.

[bnason]

Thanks for the clarification, Zeuss. I scraped/appended Monster Manuals 1-3. It's like ... 2,000 NPCs. Probably overkill to worry about marrying up the tokens for everything. The library is slow enough to open with that much as is.

Which leads to my question ... Does anyone have any "best practices" around what to parse? My game is going to be for some new people. I'm thinking of parsing Player's Handbook 1 (powers, classes, races), the Adventurer's Vault (items), Monster Manual 1, and some reference material from the DM guide (diseases, poisons). It will realistically be plenty of content for me to have available, and in the Library for them to access as needed (especially powers / feats, which I know they'll want to look at).

Really curious to learn more, however, about how some of you are using the Parser and what you build out with it.


Lastly -- some specific Qs:

1) NPCs - Fluff File - what is this in reference to (and where do you get it from?)
2) NPCs - Templates - is this from the DM guides (presumably?)
3) Reference - Manual - what is this option?
4) Tables - I'm assuming this is just what it says (effect tables, etc.) - but where do you scrape these from? I don't see anything in the Compendium mode to scrape tables.

Also -- getting an unhandled exception error trying to use the Catalog -> Catalog mode. Beginnng text is below; let me know if it would be helpful to post the full error details:


See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.IO.IOException: The directory is not empty.

[bnason]

One more dumb Q - is it possible to scrape a full adventure module from DDI? It looks like I can get a few campaign settings, etc., in the compendium mode - but when I go to parse, I'm asked for an encounters, skill, and story file. Compendium mode doesn't appear to scrape those things. Does that mean that to utilize the Adventure section of the parser, I just need to create these respective files manually?

[EugeneZ]

One more dumb Q - is it possible to scrape a full adventure module from DDI? It looks like I can get a few campaign settings, etc., in the compendium mode - but when I go to parse, I'm asked for an encounters, skill, and story file. Compendium mode doesn't appear to scrape those things. Does that mean that to utilize the Adventure section of the parser, I just need to create these respective files manually?

No, the compendium doesn't have that information, so it's impossible for the scraper to pick it up.

As for the Catalog error, I'm probably not going to fix that for a while unless people are really interested in using it. To my knowledge, no one has actually found the catalog to be useful (correct me if I'm wrong). At least some part of the original goal with that was to assist with my ill-fated character importer.

[EugeneZ]

Version 4.1.2.1 was uploaded today. Use the download links in my original post to download it. The only issue addressed was the publishedIn thingie. @arotter's login issue was not tackled since I have no way of reproducing it without communicating with him.

@DrZeuss: I was unable to reproduce seeing the publishedIn tag with items. I saw it with Powers, and for good measure, made the program look for it in nearly all cases, since it wouldn't hurt anything, but I couldn't actually reproduce it for items so I can't be 100% certain I fixed. If you could tell me what book/item you're seeing it on, I'll take another look. In any case, I hope 4.1.2.1 will take care of it everywhere.

[Zeus]

@DrZeuss: I was unable to reproduce seeing the publishedIn tag with items. I saw it with Powers, and for good measure, made the program look for it in nearly all cases, since it wouldn't hurt anything, but I couldn't actually reproduce it for items so I can't be 100% certain I fixed. If you could tell me what book/item you're seeing it on, I'll take another look. In any case, I hope 4.1.2.1 will take care of it everywhere.

I think it was either Heroes of Shadows or Heroes of the Forgotten Kingdoms, can't recall which one now. Either way its not a big deal to get rid of them with a global search and replace, however thanks for addressing the issue with the update. Much appreciated.

[EugeneZ]

Thanks. I took a look at those books in 4.1.2.1 and they no longer show the token.

[zapoqx]

I'm curious. Is it possible to add a Scrape feature for the compendium to make it batch for things that you want it to look through (say like Dragon magazine ### through ###)?
Also, is it possible to add a feature that if it says "Failed to download data" or something to attempt to re-download at least once more at the end of the scrape section process?

[EugeneZ]

@zapoqx: Both excellent ideas. I'll put both on the TODO list... shouldn't even be very hard to do.

[zapoqx]

Cool. And I feel kinda bad for not mentioning how awesome it is that you're continuing the work that was left off that was amazing in the first place. Glad to see it still going as great as ever!

[DaffyRoth]

Zapoqx, fancy meeting you here!

Anyways I have a few questions concerning how to parse some files. Firstly is there a way to make level increases to powers appear as their own line. For example the Healing Word power it increases at Level 6 to 2D6 HP regained. Whenever I put it in the parser as its own line with the power it ends up running together, or making one long sentence. I hope that makes sense.

Secondly is there an outline how to set up Poisons and Diseases? A template would be greatly appreciated.

[zapoqx]

So, I wanted to report on one I did find the "publishedin" tag and it was on the Traps section in the Dragon Annual 09.
To edit in, also the dragon magazines somewhere between 364-389 had it with the traps.
And 390-399 had an issue with something weird about the p tag not matching with the end tag of th. I looked and it weirdly had something involving <p> as the start, then later on, </th><th>. I assume Table Header, but the point is, there was no starting th tag. Don't know how to go about it right now with that, so I'll skip it (It only has cages for the most part). This th occurred again except I realized it this time it was on the equip section which had the Mount as the header. This was under the Dark Sun Campaign Setting. I suspect this has to do with the compendium not grabbing them correctly.

One more Edit:
I'm having Issues with martial Powers 2. Its something weird I can't figure out.



7/24/2011 6:22:28 PM : ERROR:System.Xml.XmlException: The '<' character, hexadecimal value 0x3C, cannot be included in a name. Line 11411, position 71.

<flavor type="formattedtext"><p><i></i></p></flavor>
<prerequisite type="string">Fighter or warlord; proficiency with any two-handed polearm or two-handed spear</prerequisite>
<effect type="string">You gain a +2 feat bonus to Intimidate checks. If you are attacking with a two-handed polearm or a two-handed spear and you have proficiency with that weapon, you also gain the following benefit. When you hit an enemy with a power associated with this feat, you can push that enemy 1 square. <br />Associated Powers: Cleave, Viper’s Strike<br /></effect>

</featLonghandStudent>
*** <featSetupDeadlyReturnsGreatDragonWarCryHeadstrongB raveryHoldTheLine<br_>>
<name type="string">Setup, Deadly Returns, Great Dragon War Cry, Headstrong Bravery, Hold the Line <br /></name>
<source type="string">Feat </source>
<description type="formattedtext"><table><tr><td><b>Prerequisite:</b>Warlord, Longhand Student feat</td></tr></table><table><tr><td><b>Benefit:</b>If you are attacking with a two-handed polearm or a two-handed spear and you have proficiency with that weapon, you gain the following two benefits. When you hit an enemy with an attack power, you gain a +1 feat bonus to AC until the end of your next turn. When you hit an enemy with a power associated with this feat, you can push that enemy 2 squares. <br />Associated Powers: Battle On,</td></tr></table></description> <class type="string">Feats</class>
<level type="number">-1973</level>
<school type="string">Greater Style</school>
I haven't found an extra carrot in any of the original txt files. All I got out of this is its around the Longhand Student section.

Edit once more...
I found it. Geez that was stupid and a pain. Apparently the following code borks it:


ZXCV123 Longhand Bravo [Greater Style]

Prerequisite: Warlord, Longhand Student feat
Benefit: If you are attacking with a two-handed polearm or a two-handed spear and you have proficiency with that weapon, you gain the following two benefits.
When you hit an enemy with an attack power, you gain a +1 feat bonus to AC until the end of your next turn.
When you hit an enemy with a power associated with this feat, you can push that enemy 2 squares.
<br />Associated Powers: Battle On, Brutal Setup, Deadly Returns, Great Dragon War Cry, Headstrong Bravery, Hold the Line<br />
The culprit is the "Hold the Line" part because its reading "Hold the Line<" it seems. However, you have to delete that entire <br /> for it to clear or it refuses to get past that point.

I think I'm done for the night with this.
Monster Manual 1 and 3. I can't seem to get the npcs to import which I'm guessing has to do with the way the monsters are currently scraped.

Here is the example I get from MM3 and the error type is the same in 1:


7/24/2011 9:27:21 PM : ERROR:System.Xml.XmlException: The 'h1' start tag on line 16013 position 49 does not match the end tag of 'languages'. Line 16013, position 74.

<powertype type="string">R</powertype>
<mm3mode type="number">1</mm3mode>
</id-00005>
</powers>
<alignment type="string">chaotic evil</alignment>
*** <languages type="string">Common, Deep Speech<h1 class="atwillpower"></languages>
<skills type="string">Athletics +9, Stealth +13</skills>

<strength type="number">10</strength>
<constitution type="number">17</constitution>
<dexterity type="number">18</dexterity>

Alright. Thats it for me.

[ShaneB]

@zapoqx: That last one is easy, its missing the close of the xml tag.. it should be <h1 class="atwillpower" /> (though I would hazard a guess that the whole h1 can be removed....)

[Dakadin]

@zapoqx:

For the first issue with the Dark Sun Campaign Setting parse it is missing the following tags <table><tr><th>. Just replace Speed per Hour for each of the 6 mounts in the equip.txt file with <table><tr><th>Speed per Hour.

For the third issue with Monster Manuals 1 and 3 you need to remove the following <h1 class="atwillpower">. It seems to append to a few of the Languages entries.

That should help you to get them parse them and hopefully EugeneZ can figure out a way to get around the issues.

[zapoqx]

Cool! Thanks. Ya, I just still wanted this reported to see if it can even be fixed.
EDIT:
GAHH! I get to the last bit and I find an issue I REALLY can't figure out and its probably smack in front of me. This is with Dungeon Annual.


7/25/2011 1:44:41 PM : Error data:
Whirling Blades Automaton Level 6 Skirmisher
Small natural animate (blind, construct) XP 250
Initiative +9 Senses Perception +1; blindsight 10
HP 66; Bloodied 33
AC 20; Fortitude 18, Reflex 20, Will 16
Immune disease, knocked prone, poison
Speed 5
C Whirling Blades (standard, at-will)
ZZX +10 vs AC; 1d6 + 4 damage, and the automaton can shift 1 square for each target it hits up to 3 squares. Close burst 1
C Remove Mark (minor, at-will)
7/25/2011 1:44:41 PM : ERROR: Fatal Parse error on: Whirling Blades (standard; at-will) ZZX +10 vs AC; 1d6 + 4 damage, and the automaton can shift 1 square for each target it hits up to 3 squares. Close burst 1


Technically this is a 2 part issue.
1: While the Parser is still open, it seems it keeps the txt file in use. Because of this, you have to write down the error and close the parser before it can be writable again.
2: I don't see what is the issue here. Here is the exact text I have.


Whirling Blades Automaton Level 6 Skirmisher
Small natural animate (blind, construct) XP 250
Initiative +9 Senses Perception +1; blindsight 10
HP 66; Bloodied 33
AC 20; Fortitude 18, Reflex 20, Will 16
Immune disease, knocked prone, poison
Speed 5
C Whirling Blades (standard, at-will)
ZZX +10 vs AC; 1d6 + 4 damage, and the automaton can shift 1 square for each target it hits up to 3 squares. Close burst 1
C Remove Mark (minor, at-will)
ZZX Close burst 5; targets an enemy marking the automaton; +9 vs Will; the mark ends.
C Bladed Destruction ( when the automaton drops to 0 hit points, )
ZZX +10 vs AC; 1d10 + 2 damage. The whirling blade automaton explodes, launching blades in all directions. Close burst 5
Alignment Unaligned Languages -
Str 14 (+5) Dex 19 (+7) Wis 6 (+1)
Con 10 (+3) Int 2 (-1) Cha 2 (-1)

Am I missing something in the lines?

[Trenloe]

Technically this is a 2 part issue.
1: While the Parser is still open, it seems it keeps the txt file in use. Because of this, you have to write down the error and close the parser before it can be writable again.
2: I don't see what is the issue here. Here is the exact text I have.

1. I can open the text file, save it etc while I have the parser open. What app are you using to edit/save the file? I'm just using Windows Notepad - nothing special.
2. You need to move the range of the power to be the first entry in the power text:


C Whirling Blades (standard, at-will)
ZZX Close burst 1; +10 vs AC; 1d6 + 4 damage, and the automaton can shift 1 square for each target it hits up to 3 squares.
C Remove Mark (minor, at-will)
ZZX Close burst 5; targets an enemy marking the automaton; +9 vs Will; the mark ends.
C Bladed Destruction ( when the automaton drops to 0 hit points, )
ZZX Close Burst 5; +10 vs AC; 1d10 + 2 damage. The whirling blade automaton explodes, launching blades in all directions.

That then works for me and it can parse. Hope it works for you! :)

[EugeneZ]

Wow, looks like there are quite a few bugs remaining. Thanks for the detailed reports, folks, I'll work on knocking these out. I'm on vacation this week, so don't expect to see a build until two weekends from now at the earliest...

[zapoqx]

1. I can open the text file, save it etc while I have the parser open. What app are you using to edit/save the file? I'm just using Windows Notepad - nothing special.
2. You need to move the range of the power to be the first entry in the power text:
C Whirling Blades (standard, at-will)
ZZX Close burst 1; +10 vs AC; 1d6 + 4 damage, and the automaton can shift 1 square for each target it hits up to 3 squares.
C Remove Mark (minor, at-will)
ZZX Close burst 5; targets an enemy marking the automaton; +9 vs Will; the mark ends.
C Bladed Destruction ( when the automaton drops to 0 hit points, )
ZZX Close Burst 5; +10 vs AC; 1d10 + 2 damage. The whirling blade automaton explodes, launching blades in all directions.That then works for me and it can parse. Hope it works for you! :)
Thanks for that reply. Got past that part. Had one more and I searched for the key part about the placement of the attack. It was on the Calmachia level 9 solo where it did the same think with its 2 Reach 2 attacks and the Close Blast 5 attack.

Well I was saying it was a 2 part issue because this happens after I receive the error. I had to literally close the parser to let it save.
I use things between GetDiz and Notepad++. Both reported not being savable and doing a little test, I couldn't delete the file as it said the 4E Parser was still using the file until the program was closed.

Upon testing, it seems the perception is showing up in the wrong location using the default unmodified scrapes. I had to look into this further. I don't exactly get it myself. It seems it shows the Perception in the senses, but not in the perception field. Sometimes it will show in the senses the perception bonus, but still not show the perception in the perception field.

And no problem EugeneZ.

EDIT: After getting back info and looking into it further, here is the best way to explain:
https://bit.ly/mQ6QkS - Without personalities
https://bit.ly/pveMDV - With Personalities

Observe the perception. These come from:
Murklord Frog - Monster Manual 3
Kestrekel Carrion Eater - Dark Sun Creature Catalog
Ice Warrior Raider - Dungeon Annual

Seems like Ice Warrior Raider is the only one willing to get a perception check correct. I suppose if its not an issue, I'll post what I have listed for those to see if we can fire out what the parser is doing during compendium download. If its an issue mods, feel free to edit this post and remove the code tags



Murklord Frog Level 2 Skirmisher
Medium natural beast (aquatic) XP 125

HP 39; Bloodied 19 Initiative +7
AC 16, Fortitude 13, Reflex 14, Will 13 Perception+4
Speed 6, swim 6 Low-light vision


Traits
X Aquatic
ZZX The murklord can breathe underwater. In aquatic combat, it gains a +2 bonus to attack rolls against nonaquatic creatures.

Standard Actions
m Bite ? At-Will
ZZX Attack: Melee 1 (one creature); +7 vs. AC
Hit: 1d6 + 5 damage, or 1d12 + 5 while the murklord is bloodied.
M Leapfrog ? Recharge ? ? ?
ZZX Effect: The murklord shifts 3 squares, makes the following attack, and shifts 3 squares again.
Attack: Melee 1 (one creature); +5 vs. Reflex
Hit: 2d6 + 4 damage, or 2d12 + 4 while the murklord is bloodied.

Minor Actions
M Tongue Lash ? At-Will
ZZX Attack: Melee 2 (one creature); +5 vs. Reflex
Hit: The murklord slides the target 1 square.
Skills Acrobatics +10
Str 16 (+4) Dex 18 (+5) Wis 16 (+4)
Con 15 (+3) Int 4 (-2) Cha 8 (0)
Alignment unaligned Languages -

-----------

Kestrekel Carrion Eater Level 1 Minion Skirmisher
Tiny natural beast XP 25

HP 1; a missed attack never damages a minion. Initiative +4
AC 16, Fortitude 12, Reflex 14, Will 13 Perception+6
Speed 2, fly 6


Standard Actions
m Claw ? At-Will
ZZX Attack: Melee 0 (one creature); +6 vs. AC
Hit: 4 damage, or 5 against a bloodied target.
M Darting Rake ? At-Will
ZZX Effect: The kestrekel shifts 3 squares and uses claw.
Str 8 (-1) Dex 15 (+2) Wis 12 (+1)
Con 11 (0) Int 1 (-5) Cha 6 (-2)
Alignment Unaligned Languages -

--------

Ice Warrior Raider Level 2 Soldier
Medium elemental animate (cold) XP 125
Initiative +5 Senses Perception +1
HP 42; Bloodied 21
AC 17; Fortitude 15, Reflex 13, Will 13
Immune disease, poison; Resist 10 cold
Speed 6 (ice walk)
m Maul (standard, at-will) ? Cold
ZZX +9 vs AC; 1d8+3 damage, and the target is slowed (save ends). Against a slowed target, the attack deals 1d6 extra cold damage.
M Ice Embrace (minor, at-will) ? Cold
ZZX +7 vs Fortitude; the target is immobilized by ice (save ends).
Alignment Chaotic evil Languages Primordial
Str 17 (+4) Dex 14 (+3) Wis 11 (+1)
Con 18 (+5) Int 11 (+1) Cha 11 (+1)
Equipment: maul.

[Zeus]

@ zapoqx: Try adding a whitespace character between the keyword Perception and the + modifier for the first two NPCS e.g.

Perception +4

And see if that parses, OK. From looking at the Compendium the data entries for a Murklord Frog and Kestrekel Carrion Eater, they both don't have the whitespace that the Parser expects.

[DaffyRoth]

Heh dont know if its possible, but WOTC changed the disease stat block. Was wondering if it would be possible to get it changed in the Parser to reflect that.
If not there are always alternatives lol.

And to Zapoqz, ya the perception scores need to be on the same line as initiative in order for it to work as intended.

[zapoqx]

so... to work where it will let me do the double click of that perception roll, it needs to have that white space...
And work as intended?

[Zeus]

@zapoqx: Let me know if the NPC entries below parse OK. As joseph.pimpo has pointed out, in addition to the whitespace, Sense Perception text needs to be on the same line as the Initiative entry. See below.


Murklord Frog Level 2 Skirmisher
Medium natural beast (aquatic) XP 125
Initiative +7 Senses Perception +4
HP 39; Bloodied 19
AC 16, Fortitude 13, Reflex 14, Will 13
Speed 6, swim 6 Low-light vision
Traits
X Aquatic
ZZX The murklord can breathe underwater. In aquatic combat, it gains a +2 bonus to attack rolls against nonaquatic creatures.
m Bite ? At-Will
ZZX Attack: Melee 1 (one creature); +7 vs. AC
Hit: 1d6 + 5 damage, or 1d12 + 5 while the murklord is bloodied.
M Leapfrog ? Recharge ? ? ?
ZZX Effect: The murklord shifts 3 squares, makes the following attack, and shifts 3 squares again.
Attack: Melee 1 (one creature); +5 vs. Reflex
Hit: 2d6 + 4 damage, or 2d12 + 4 while the murklord is bloodied.
M Tongue Lash ? At-Will
ZZX Attack: Melee 2 (one creature); +5 vs. Reflex
Hit: The murklord slides the target 1 square.
Skills Acrobatics +10
Str 16 (+4) Dex 18 (+5) Wis 16 (+4)
Con 15 (+3) Int 4 (-2) Cha 8 (0)
Alignment unaligned Languages -

-----------

Kestrekel Carrion Eater Level 1 Minion Skirmisher
Tiny natural beast XP 25
Initiative +4 Senses Perception +6
HP 1; a missed attack never damages a minion.
AC 16, Fortitude 12, Reflex 14, Will 13
Speed 2, fly 6
m Claw ? At-Will
ZZX Attack: Melee 0 (one creature); +6 vs. AC
Hit: 4 damage, or 5 against a bloodied target.
M Darting Rake ? At-Will
ZZX Effect: The kestrekel shifts 3 squares and uses claw.
Str 8 (-1) Dex 15 (+2) Wis 12 (+1)
Con 11 (0) Int 1 (-5) Cha 6 (-2)
Alignment Unaligned Languages -

[Mooses8D]

Ran into a few errors while trying to parse a few of the magazines, specifically #373. Here's the error:


7/29/2011 2:45:42 PM : ERROR: line:Miss: Half damage, and the target is dazed (save ends).
7/29/2011 2:45:42 PM : Start Power Parse
7/29/2011 2:45:42 PM : Reference parameters set
7/29/2011 2:45:42 PM : NPCs parameters set
7/29/2011 2:45:42 PM : Items parameters set
7/29/2011 2:45:42 PM : Adventure parameters set
7/29/2011 2:45:42 PM : Root parameters set
7/29/2011 2:45:42 PM : Subtree Verified
7/29/2011 2:45:42 PM : Top level verified
7/29/2011 2:45:42 PM : Starting parse


It doesn't even get far into it and it pops up with an error I've never seen before.


The 2nd issue I ran into occurred while I was trying to compile all the Dragon Magazine information into 1 parse. I would take the information from the text documents, and put them all into 1 so they would make 1 large module. The error I keep getting is this:


7/29/2011 2:51:02 PM : ERROR:System.Xml.XmlException: The '[' character, hexadecimal value 0x5B, cannot be included in a name. Line 3156, position 741.

<type type="string">Feature</type>
<flavor type="formattedtext"><p><i>You can spend an action point to reroll an attack roll against a dragon, or to end a condition currently affecting you that originated from a dragon, and that a save can end. Using this feature replaces the normal action point benefit of taking an extra action.</i></p></flavor>
<stringflavor type="string">You can spend an action point to reroll an attack roll against a dragon, or to end a condition currently affecting you that originated from a dragon, and that a save can end. Using this feature replaces the normal action point benefit of taking an extra action.</stringflavor>

</featureDragon-SlayersAction_178>
*** <featureInstinctFeatUtilityWithYourDefiantSnarlTheA dvantageYourFoesHadGainedEvaporatesGivingYouAnOpen ing_EncounterFreeActionPersonalTrigger_YouAreFlank edOrYourPreyGainsCombatAdvantageAgainstYouEffect_Y ouDoNotGrantCombatAdvantageToThoseWhoTriggeredThis PowerUntilTheEndOfYourNextTurn_IfYourPreyIsAmongTh oseWhoTriggeredThisPowerYouGainCombatAdvantageAgai nstYourPreyUntilTheEndOfYourNextTurn_BravosFinishF eatUtilityIfYourViciousAssaultDoesntBringYourPreyD ownItSurelyShowsYourPreyWhatTheNearFutureHolds_Dai ly_ReliableStandardActionSpecial_InAdditionToTheKe ywordAboveThisPowerIsConsideredToHaveTheSameKeywor dsAndRangeAsTheAt-willPowerYouUseWithIt_Target_OneCreatureThatIsYour PreyEffect_UseAnAt-willAttackPowerOnTheTarget_IfTheAttackHitsYouDeal2[W]ExtraDamageOr_2DiceOfDamageIfItIsANonweaponAttackA ndTheTargetIsDazed_saveEnds__Level19Swap_3[W]ExtraDamageOr_3DiceOfDamage_Level29Swap_3[W]ExtraDamageOr_3DiceOfDamageAndTheTargetIsStunnedUn tilTheEndOfYourNextTurn_>
<name type="string">Instinct Feat
<recharge type="string">Encounter</recharge>
<keywords type="string">Weapon</keywords>
<action type="string">Standard Action</action>
<range type="string">Personal</range>


It seems to occur on a power that's actually within issue #373. Any suggestions?


---===Edit===---

Took out a large chunk of the descriptive text so it wouldn't be so massive. Sorry about that.

[zapoqx]

Well I got it to work fine with just the white space and it seems to work fine with those not using the "Senses Perception" thing next to initiative and with. It seems to me the only difference with the 2 is that without placing perception in the same line as initiative, it seems to just make the card wider. Otherwise, it seems to work just fine.
@Moose: I didn't have an error occur with 370-379 (I compiled mine that way by appending from Dragon magazine 370 all the way to 379). I don't think I had one at least. As to the 2nd error, that might be because of the parse it took from Compendium to text. I had an issue like that before and it was because of a simple apostraphe that in NotePad++, it never caught it, but when I changed the text format to whats used in GetDiz, the apostraphe was a different symbol character. Just simply go through and check what special characters there are in that section and replace it with the regular ascii key needed.

[khersheyjr]

I can't seem to figure out what I need to get this one to work...its for the Dark Sun Campaign setting.

7/30/2011 12:31:14 AM : ERROR:System.Xml.XmlException: The 'p' start tag on line 1653 position 5 does not match the end tag of 'th'. Line 1653, position 242.

<weight type="number">0</weight>
<cost type="number">225</cost>
<type type="string">Mount</type>
<subtype type="string"></subtype>
<description type="formattedtext">
*** <p><p>A crodlu is a large, flightless drake with a beak and weak, clawed forelimbs that can be used to manipulate small objects. It is a tough and aggressive hunter in the wild. When trained, it makes an excellent war mount. Speed per Hour</th><th>Speed per Day</th><th>Normal Load (lb.)</th><th>Heavy Load (lb.)</th><th>Push/Drag (lb.)</th></tr><tr><td>4 miles</td><td>40 miles</td><td>190</td><td>380</td><td>950</td></tr></table></p></p>
</description>
</crodlu>
<warcrodlu>
<name type="string">War crodlu</name>
<weight type="number">0</weight>

The block on scraped code is:
ZHead; Mount
ZDesc; <p>A crodlu is a large, flightless drake with a beak and weak, clawed forelimbs that can be used to manipulate small objects. It is a tough and aggressive hunter in the wild. When trained, it makes an excellent war mount. Speed per Hour</th><th>Speed per Day</th><th>Normal Load (lb.)</th><th>Heavy Load (lb.)</th><th>Push/Drag (lb.)</th></tr><tr><td>4 miles</td><td>40 miles</td><td>190</td><td>380</td><td>950</td></tr></table></p>
ZItem; Crodlu; 225 gp; -
ZDesc; ;

prolly something simple...thanks for any help

[Zeus]

Try the block below, your original block was missing opening tags for the table entries and a closing tag for the paragraph:


ZHead; Mount
ZDesc; <p>A crodlu is a large, flightless drake with a beak and weak, clawed forelimbs that can be used to manipulate small objects. It is a tough and aggressive hunter in the wild. When trained, it makes an excellent war mount.</p><table><tr><th>Speed per Hour</th><th>Speed per Day</th><th>Normal Load (lb.)</th><th>Heavy Load (lb.)</th><th>Push/Drag (lb.)</th></tr><tr><td>4 miles</td><td>40 miles</td><td>190</td><td>380</td><td>950</td></tr></table>
ZItem; Crodlu; 225 gp; -
ZDesc; ;

[ShaneB]

@khersheyjr: its the first <th> that is missing before "Speed per Hour"

[khersheyjr]

Thanks for the help. I am still getting more errors with the Dark Sun Campaign Settings parse but thanks to you all I can see what I need to fix :).

[ShaneB]

If you have the specs download eclipse (from eclipse.org) it has a great XML editor that you can use to parse portions of the document and will show where there are errors.

[khersheyjr]

Ill check that out. Thanks

[primarch]

Hi!

DrZeuss, as I have updated my books with you new parser exe. I has noticed that when you scrape divinity feats that have a related "feat power" it appears in the text file as a "feat utility".

When you parse such things you can't get the program these "feat utilities" as "feat powers" and these abilities don't show up in the finished module. Regardless if you have the "feat power to library" on "yes".

The work around I used to fix this was to use Notepad++ to fins each divinity feat with an associated feat power and change the word "utility" i the heading to "power". This permitted the feat power to be parsed and show up correctly.

This issue was specifically the scrape from Divine Power. It may or may not be an issue with other books.

I guess at some point the parser needs to recognize "feat utility" as "feat power" to avoid the manual process I previously described to get them done correctly in the module.

Thanks for updating the parser, I haven't been this busy in over a year!

Primarch

[Zeus]

@Primarch: Sorry old bean but you have your wires crossed. I didn't write nor do I maintain the 4E Parser. Tenian developed the original tool and Eugenez is now maintaining the development.

Hopefully Eugenez can address your issue.

[primarch]

@Primarch: Sorry old bean but you have your wires crossed. I didn't write nor do I maintain the 4E Parser. Tenian developed the original tool and Eugenez is now maintaining the development.

Hopefully Eugenez can address your issue.

Hi!

Oh, my bad, I use so many of your extensions and that wonderful 4e decompiler that I thought you did the new exe update too. :o

Hopefully Eugenez will pop up and give the question a crack. Since my original post I have noticed this with other books too. The PHB1 scrape has an interesting quirk were the ZPOWER line of the channel divinity refers to the next feat after it and not the actual feat power! Weird.

Easily fixed though by replacing the faulty reference to the correct one.

Having updated and re-parsed all the books (including a massive compendium I made that includes all dragon and dungeon magazine entries), the updated parser is working very well, with little or no hiccups.

The feat power are the only thing I noticed that was off.

Many thanks to Eugenez for maintaining it, my old books were very dated until now. :)

Primarch

[EugeneZ]

Just got back from a pleasant vacation in Toronto. Thanks for the reports, guys. I'm compiling a list of issues based on this stuff and I hope to do a release this weekend with at least some of the fixes.

[Crispyrat]

Hello,

Having issue with the Dark Sun Campaign parse. Got past the mount issue, and the scrape parses to a module, but I can't open it (swipe the book open) in FGII. When that issue arises (the book won't open), what is this generally indicative of?

I have parsed this as a db.xml and a client.xml.

Also, I had a typo that was driving me bonkers, so I deleted the module, renamed and reparsed it. It shows fine, but the old name (now defunct) still shows. How do I get rid of that? I imagine it is a file I need to whack - but which one?

Thanks!

-Chris

[Crispyrat]

Found my issue. The module next to it was elongated due to the creator text being obtrusively long, which meant that I couldn't "touch" the Dark Sun book to open it. Moved the window around a bit, and could open the book.

Thanks.

[Teldil]

Any reason why I can't seem to be able to scrape the ITEMS data from Player's Handbook 1 ?
When it reaches "setting up items" It always crashes the program with a "(500) Internal Server Error" error.

Rest of the stuff except items scrapes with no problem....

Thanks for the help.

[Dakadin]

It sounds like you are using an old version of the parser. The latest one updated by EugeneZ resolves that issue.

[Teldil]

Hmm I'm using 4.0.118 , isnt this the latest version? Should I be using the beta ?

[Dakadin]

No that isn't the latest version. You can get the latest one on the first post of this thread and it is the beta.

[Teldil]

On the first post of this thread that IS the latest version....

[Dakadin]

Yes, that is the latest version that Tenian wrote. He has stopped developing it and EugeneZ took over and resolved the item issue along with a few other things. I think he is listing it as a beta version until resolves a few more issues.

[Teldil]

AH I got it now....I thought the beta was optional and I usually prefer final versions so I got the stable version only....

I'll update to the beta one, thanks! :)

[Teldil]

Not sure if this is the right thread for this....but dragging a power from a module to my character sheet's powers page doesn't seem to do anything.

In the guide it says I should only drag and drop the power into the powers page, but nothing happens when I do that.

Dragging and dropping equipment from the same module works....what am I missing ?

[Dakadin]

I just tried every combination I could think of and the only way it didn't work if it was dropped on the powers section of the character sheet was if I was using the mini sheet. Can you give a bit more detail of what power you are trying to add and how you are doing it?

[Teldil]

Hmm...well I'm not on the mini sheet.

It doesnt matter what power I'm trying to add, every power I drag from the library to the powers page does nothing. Nothing is being added to the powers page.

I'm on the char management, using a the Primal Power book I created as a client module, and I'm dragging the red dragon symbol next to each power over to the power page on the char sheet. Am I doing it wrong maybe?

[Dakadin]

It sounds like you are doing it right to me. I think you will need someone with more experience than I to help you with this issue.

[zapoqx]

I just checked to see if I could figure it out.
Are you trying to do it in the manage characters section of FG2? If so, then ya, I can't seem to either. HOWEVER, if I go to my test campaign and have the character saved, open it up, then I can add the powers from the library, it works.

[Dakadin]

Teldil, I see you already found this but I figured I would post it in case anyone else was having the same issue.

This is a known issue in 2.8 when using the Manage Characters option. Here is a link to the thread: https://www.fantasygrounds.com/forums/showthread.php?t=15087

[Teldil]

Yes I was trying to do it via the Manage Chars section. and yes I have already found out this is a known bug and actually am using the beta 2.8.1 now and it works there ;)

Thanks Dakadin and zapoqx for the help! :)

[DaffyRoth]

Ok heres my question. Is there away to parse fantastic terrain?

[Zeus]

Ok heres my question. Is there away to parse fantastic terrain?

What's fantastic terrain? Not familiar with that one.

[DaffyRoth]

Heres an example from Dark Sun:

Settled Ash SingleUse Terrain
Ash from plants destroyed by defiling magic coats the around.
The slightest disturbance kicks up an obscuring cloud.
Standard Action
Requirement: You must be adjacent to a square of settled
ash.
Check: Athletics check (easy DC) to raise a cloud of ash.
Success: The ash fills the air, obscuring vision.
Target: Each creature in a close blast 3
Attack: level + 3 vs. Fortitude
Hit: The target is blinded until the end of your next turn.
Effect: The area of the blast is heavily obscured until the end
of the encounter.

I belelieve that terrain was added in with the DMG2. It acts like traps but, its not really a trap per-se. So ya, if there is a way to get these to work that would be great.

Also I am currently working on a way to get themes to work. So far I have just made them their own class.

[DaffyRoth]

Also is there a way for the Parser to handle multiple triggers on a trap? So far it seems to just relist all the attacks or they dont show up at all. Again Dark Sun CC is the reference for that.

EDIT: Also, I am having issues with the ZLINK in the reference manuals. I am trying to make one point to an NPC. This is the string I am unsing:

ZLINK npc;silt runner;Dark Sun Creature Catalog

All it does is point to an empty npc stat block. I am read through the parser doc and from what I can tell I am doing it right.....I dunno though.

[primarch]

Also is there a way for the Parser to handle multiple triggers on a trap? So far it seems to just relist all the attacks or they dont show up at all. Again Dark Sun CC is the reference for that.

EDIT: Also, I am having issues with the ZLINK in the reference manuals. I am trying to make one point to an NPC. This is the string I am unsing:

ZLINK npc;silt runner;Dark Sun Creature Catalog

All it does is point to an empty npc stat block. I am read through the parser doc and from what I can tell I am doing it right.....I dunno though.

Hi!

I can confirm the issue with the ZLINK pointing to an empty npc stat block. I have yet to find the right coding to fix it though.

Primarch

[DaffyRoth]

Ok I figured it out by digging through the old parser updates. So with ZLINK, you can use libnpc or libtrap and that links to the library for both traps and npc.

SO....it looks like this:

ZLINK libnpc;Big Bad Boss

I am pretty sure that the name has to match the same name as what you have in said library.

So ya.....yay!

[DeviousAlpha]

I hope I have this understood wrongly but I'll ask and expect the worst.

For -each- book, I must manually copy and paste the Stat boxes into an ASCII file. Then edit (fix) the stat boxes, and assign groups and such. Then following this I must do in a seperate file all the fluff parts. And then I use the parser to scrape the data into mod files, paste them into the correct directory and its done.

Forgive me for sounding a little blown away but, that sounds like a lot of work with so many books!

Have I got this wrong?

[EugeneZ]

You don't need to "fix" the stat boxes, but you do need to annotate them so that the parser can understand where a power begins and ends. There is no easy automated way of doing this, so you need to help the parser.

Keep in mind that if you have a DDI subscription, the scraper will create the parse files for you, making life easier. (But you obviously can't do this for things the Compendium does not have, such as the fluff text.)

It is a lot of work, but a number of people here have done it anyway. What is your proposed alternative?

[DeviousAlpha]

You don't need to "fix" the stat boxes, but you do need to annotate them so that the parser can understand where a power begins and ends. There is no easy automated way of doing this, so you need to help the parser.

Keep in mind that if you have a DDI subscription, the scraper will create the parse files for you, making life easier. (But you obviously can't do this for things the Compendium does not have, such as the fluff text.)

It is a lot of work, but a number of people here have done it anyway. What is your proposed alternative?

I have no idea for an alternative, just wanted to check this is the case before I dived into more work than I had to. It's no biggie, when writing up campaigns in the past I've usually taken little cards and written down the monsters on them, save lugging all the books about. Its basically the same thing only I only have to write them down once, I can never lose them (unless I lose the file, oh gods!) and I can even just drag and drop them into future campaigns.

To the workbench I go!

[EugeneZ]

The Parser is kind of complicated and a bit awkward at times. What seems like an extremely daunting task at first becomes easier as you get into a pattern for annotating the text files. Plus, you can decide for yourself just how much information you wish to put into the parser... I'm sure the majority of users don't bother with the fluff, for example. Or even the more esoteric blocks, like diseases, which are only handy once in a while. I think most people do the powers and the NPCs and stop there. Powers are fairly easy once you figure out the setup of the classes/races files, since they require no markup. NPCs require a bit, but like I said, you'll fall into a pattern.

That said, there are two other options you might want to consider:

1) Paying for a single month of Compendium access allows you to use the scraper to scrape hundreds of sources (books, adventure, and magazines) produced by WotC at once. Pay for it for one month, do the large books in the first week, and then spend a few days scraping the smaller stuff. That doesn't get you the fluff and some other stuff, but it gets you 90% of the way there.

2) This suggestions assumes you own all the books you want to make mods for. If you do indeed own them, I don't know what the legality of this suggestion is (not say it's NOT legal... I just don't know if it is), but you may try and find a person who has already used the parser to create the modules and get them from him. Frankly, most of the players/DMs I know took this path... when WotC was actively releasing books, one or two people who happened to have the time would parse the book and then distribute to friends, who presumably also owned the books. They gave it to their friends, and so on.

[DaffyRoth]

EugeneZ, how do you set up traps for mutliple triggers on a trap? And is there away for the parser to figure out fantastic terrain? Also it seems that the trigger for a trap gets rolled under the perception for whatever reason, instead of getting its own line.

[EugeneZ]

The second trigger needs to say "Secondary Trigger" instead of just "Trigger".

As for fantastic terrain, I can't see any code in the parser that does this. However, some types of stat blocks are handled by other similar stat blocks... can't really think about any that might fit terrain, though. Does the 4e ruleset even have a means of displaying it? If not, the parser probably doesn't parse it.

[DaffyRoth]

I was thinking about running it as a trap. I dunno I suppose its not that big of a deal. BTW is there an updated trap template? Like I said the one that is included in the parser doc, doesnt give Trigger its one line. It puts in the text for perception. Its kind weird looking.

[Zeus]

Just to add. If your an RPGA member, you can join the Yahoo group that was setup to support RPGA online LFR events (https://games.groups.yahoo.com/group/RPGA_Group/files/FG%20II%20Campaign%20Files/). There you will find a repository of FGII modules containing the raw essentials for LFR online play.

These include powers/feats/class features/paragon paths/epic destinies from PHB, PHB2, PHB3 and the other class/race splat books, NPCs, Traps, Poisons, Diseases, Items, Magic Items from DMG, DMG2, AV and AV2. You can also find the FR Campaign and Player Guides as .mods not to mention a growing collection of modules for Dragon/Dungeon content.

The files are usually updated by members of the RPGA community following errata updates so they should be fairly up to date.

As Eugenez has stated, these essential scrapes are all that is really needed for online FGII D&D play, the rest is nice to have but not essential.

[DeviousAlpha]

Just to add. If your an RPGA member, you can join the Yahoo group that was setup to support RPGA online LFR events (https://games.groups.yahoo.com/group/RPGA_Group/files/FG%20II%20Campaign%20Files/). There you will find a repository of FGII modules containing the raw essentials for LFR online play.

These include powers/feats/class features/paragon paths/epic destinies from PHB, PHB2, PHB3 and the other class/race splat books, NPCs, Traps, Poisons, Diseases, Items, Magic Items from DMG, DMG2, AV and AV2. You can also find the FR Campaign and Player Guides as .mods not to mention a growing collection of modules for Dragon/Dungeon content.

The files are usually updated by members of the RPGA community following errata updates so they should be fairly up to date.

As Eugenez has stated, these essential scrapes are all that is really needed for online FGII D&D play, the rest is nice to have but not essential.


Thanks a bunch for the link, you have saved me countless hours of effort. I do own the books so I'm not too worried about using these ones, surely thats legal.

[GenX]

Hi DnD-Parser professionals,

i tried to generate my first DnD-4E Module for FG2.
I installed everything like EugeneZ explained it at his main article.

I think i am doing something wrong or maybe there is still a bug in the Parser.

My problem is, after i scraped the "Dragon Magazine 381" with my DnD subscriber account i get this files :
armor.txt
class.txt
equip.txt
mi.txt
npcs.txt
powers.txt
race.txt
rituals.txt
weapon.txt

After this step I switched to the Parse-window. I insert every information and txt-file I have got.
Then i say Parse and the tool generate me a "Dragon Magazine 381.mod" file. After that I opened FG2 in Version 2.8.1 and opened the new module in the modules-window and checked the result in the Libary. Until this point I thought everything has worked.

But when I check my needed feat "Amoth's Grace" I always get a problem which I can´t explain to me.

The linked Power of this feat is totaly wrong ! Normaly there should be linked the "Channel Divinity: Amoth’s Grace" Encounter Power but somehow always the "Black Wrath of Hell" Daily-Power is inserted, as you can see on my screenshot.

I haven´t any idea why this is happening.
Can someone please help me ?

https://img7.imageshack.us/img7/8745/fehlerimparser.th.jpg (https://imageshack.us/photo/my-images/7/fehlerimparser.jpg/)

[DaffyRoth]

Heh the scrapeing doesnt always return the proper information. I am am just now learning that myself.....

[primarch]

Hi!

Having done master modules that include all dragon magazine information I can tell you that ALL channel divinity powers do NOT correctly parse the corresponding power.

The problem is that the parser parses the channel divinity powers (if I remember correctly) as "insert channel divinity power name here" Utility Power.

Note it uses "utility power" NOT "feat power".

To make the powers show up appropriately you MUST manually change the "utility" word to "feat" so it reads "feat power".

Example: Amroth's grace is parsed as Amroth's Grace Utility Power it MUST be changed to Amroth's Grace Feat Power and the power will parse correctly.

In your case Genx I suspect since it does not recognize the tag Utility power for that feat power its skipping it and adding the power next in line in the text file, Black wrath of hell in your case. I observed something similar as well. Correcting the tag for the feat power will correct this.

Use a program like Notepad++ where you can easily search for the words utility power or the specific channel divinity name and make the change.

While we are on this subject I would also like to add that the parser also parses incorrectly class "features". Namely because it is parsing them under another tag like utility, so you need to manually go in an rename them with the tag of class feature otherwise they will NOT show up either after the parse.

I have encountered quite a few things like this. I suspect there may be slight conflicts with the new exe and the old program.

If Eugene needs me to list them more thoroughly at some point let me know.

Primarch

[GenX]

Ok, yesterday I have tried something similar like this without success.
So thank you so far for the detailed explaination. I'll try that tonight.

Also, I thank EugeneZ for his work so far and hope that he finds the time to improve the parser so it can handle the mentioned problems in the future.

[GenX]

Mmm, somehow it doesn&#180;t seems to work for me.

Here are the original scraped entries from the file powers.txt for "Amoth's Grace" :

from line 1487 to 1495


Amoth's Grace Feat Utility
Through Amoth’s power, you channel strength to your allies, allowing them to stand fast against evil.

Encounter ? Divine, Healing
Immediate Reaction Ranged 5
Channel Divinity: You may only use one Channel Divinity power per encounter.
Trigger: You or an ally within range of you are either subject to a critical hit or bloodied by an attack
Effect: The triggering creature can spend a healing surge.
Special: You must take the Amoth’s Grace feat to use this power.

from line 2116 to 2120


ZXCV123 Amoth's Grace

Prerequisite: Channel Divinity class feature
Benefit: You can invoke the power of your deity to use Amoth’s Grace.
ZPOWER Amoth's Grace

again this info but also from line 2275 to 2283 !?


Amoth's Grace Feat Utility
Through Amoth’s power, you channel strength to your allies, allowing them to stand fast against evil.

Encounter ? Divine, Healing
Immediate Reaction Ranged 5
Channel Divinity: You may only use one Channel Divinity power per encounter.
Trigger: You or an ally within range of you are either subject to a critical hit or bloodied by an attack
Effect: The triggering creature can spend a healing surge.
Special: You must take the Amoth’s Grace feat to use this power.

@primarch: Or have you edit the mod files ????

[primarch]

Mmm, somehow it doesn´t seems to work for me.

Here are the original scraped entries from the file powers.txt for "Amoth's Grace" :

from line 1487 to 1495



from line 2116 to 2120



again this info but also from line 2275 to 2283 !?

Hi!

First off GenX apologies for trying to do this from memory. Even the slightest omission can make for a frustrating experience! :o

I not only have the solution, but also found the "why" its taking on another power to it as well as the solution. I am pasting the proper code directly from my master files that I have tested as working.

Your version:

ZXCV123 Amoth's Grace

Prerequisite: Channel Divinity class feature
Benefit: You can invoke the power of your deity to use Amoth’s Grace.
ZPOWER Amoth's Grace

The root of all your woes start here, note the corrected version:

ZXCV123 Amoth's Grace [Divinity]

Prerequisite: Channel Divinity class feature
Benefit: You can invoke the power of your deity to use Amoth’s Grace.
ZPOWER Channel Divinity: Amoth's Grace

Several things need to be pointed out:

1. Note the "[Divinity]" this categorizes these feats separately and lists them separately. You will not need to hunt for it amongst other standard feats as they will now be in their own category.

2. Note the last line "ZPOWER". The power needs to have the "Channel Divinity: insert power name here" you do NOT need to add feat power nor feat utility here.

3. The reason why it was adding other power descriptions after the feat name is because you should NOT have POWERS after the feat entry, especially with no spaces after. For some reason if you do this the code will include the powers after in the feat description. In my case it added SIX thief powers!

I recommend you do as I did list powers FIRST and then all the "feats lasts so no powers are after feat entries. OR list the corresponding feat power after the feat with a space between them.

Your version:

Amoth's Grace Feat Utility
Through Amoth’s power, you channel strength to your allies, allowing them to stand fast against evil.

Encounter ? Divine, Healing
Immediate Reaction Ranged 5
Channel Divinity: You may only use one Channel Divinity power per encounter.
Trigger: You or an ally within range of you are either subject to a critical hit or bloodied by an attack
Effect: The triggering creature can spend a healing surge.
Special: You must take the Amoth’s Grace feat to use this power.

Corrected Version:

Channel Divinity: Amoth's Grace Feat Power
Through Amoth’s power, you channel strength to your allies, allowing them to stand fast against evil.

Encounter ? Divine, Healing
Immediate Reaction Ranged 5
Channel Divinity: You may only use one Channel Divinity power per encounter.
Trigger: You or an ally within range of you are either subject to a critical hit or bloodied by an attack
Effect: The triggering creature can spend a healing surge.
Special: You must take the Amoth’s Grace feat to use this power.

1. Note the ONLY change you need to make is in the title adding "Channel Divinity: the power name and at the end "feat power". Note you can end it in "feat utility" as well all this changes is where on the character sheet it will list the power.

2. Note you have the same power listed TWICE. I noticed this as well. Delete one, since its not necessary.

In summary the corrected format is:

ZXCV123 Amoth's Grace [Divinity]

Prerequisite: Channel Divinity class feature
Benefit: You can invoke the power of your deity to use Amoth’s Grace.
ZPOWER Channel Divinity: Amoth's Grace

Channel Divinity: Amoth's Grace Feat Power
Through Amoth’s power, you channel strength to your allies, allowing them to stand fast against evil.

Encounter ? Divine, Healing
Immediate Reaction Ranged 5
Channel Divinity: You may only use one Channel Divinity power per encounter.
Trigger: You or an ally within range of you are either subject to a critical hit or bloodied by an attack
Effect: The triggering creature can spend a healing surge.
Special: You must take the Amoth’s Grace feat to use this power.

IMPORTANT: Keep in mind that it is recommended you do NOT list powers after the feat entry that starts with ZXCV (the numbers 123 are for heroic, 456 for paragon and 789 for epic) and ends with the ZPOWER.

You master file should place all POWERS first and all FEAT entries LAST for extra safety that nothing bizarre occurs.

The powers should no list correctly under their heading "divinity feats" and when you click on Amoth's Grace you will not only get the feats description but an additional link to the feat power.

Let me know how it goes. :)

Primarch

[GMTroll]

Ho hum... looks like WotC have been messing with Items again despite Mordenkainen's not going in this month.

Since the update today I'm getting the error: (500) Internal Server Error every time the parser goes to get the Items list for a source book and only 10 more Dungeon issues to scrape for the full set

[zapoqx]

Thats because the items has completely changed from their former entrees to be more like what Mordenkainen's item entrees are. So the parser is gonna probably need an overhaul anyways with the item section.

[Zeus]

Mordenkainen's Magnificent Emporium does indeed change the stat block for Magic Items to include frequency of availability i.e. Common, Uncommon, Rare etc. etc.

e.g. the old AV armor item Armor of Durability's header changes from

Armor of Durability Level 4+

to

Armor of Durability Level 4+ Common

As far as I can tell, the remainder of the block stays the same, except for the color which has been changed from Yellow to Orange (eek).

Hopefully, its not a big change required to get the Parser scraping Magic Items again.

[GenX]

@primarch

THANK YOU VERY MUCH !
It worked !

[primarch]

@primarch

THANK YOU VERY MUCH !
It worked !

Hi!

Glad to help! :)

Primarch

[johankov]

As GMTroll, I have the same problem: (500) Internal Server Error every time the parser goes to get the Items list for a source book :(

Not being a developer, is someone going to update the 4E Parser? Or is there another way to retrieve items?

Otherwise a big thanks to those who worked on this great tool :)

[DaffyRoth]

Could it be that the parser does not recognize the Common-Rare tags?

[primarch]

Could it be that the parser does not recognize the Common-Rare tags?

Hi!

I have confirmed this. Parser does not currently recognize the rarity tags and will give a server 500 error and not complete item parsing.

On another note, does the parser recognize the new monster format?

I tried making a compendium of all dungeon monster entries and I get fatal errors on trying to make a monster module when using freshly downloaded MM entries.

Example:

Astral Shade Level 17 Solo Lurker
Large immortal humanoid XP 8000
Initiative +19 Senses Perception +11; blindsight 10, darkvision
HP 448; Bloodied 224
AC 29; Fortitude 28, Reflex 30, Will 28
Immune disease, poison; Resist insubstantial; Vulnerable force (the next attack against the astral shade ignores its insubstantial)
Saving Throws +5
Speed fly 10 (hover); phasing
Action Points 2
m Astral Slam (standard, at-will) ? Radiant
ZZX +20 vs Fortitude; 3d8 + 4 radiant damage and the target gains vulnerable 10 to radiant damage until the end of the astral shade’s next turn. If the astral shade is bloodied, the target is also restrained (save ends).
r Radiant Ray (minor, at-will) ? Radiant
ZZX One or two targets; +22 vs Reflex; 3d8 radiant damage, and the target slides 1 square. If the astral shade is bloodied, the attack instead deals 3d8 + 5 radian damage and slides the target 2 squares.
M Wrathful Strike (immediate reaction, when damaged by an attack, at-will)
ZZX The astral shade teleports 4 squares and makes an astral slam attack or a radiant ray attack.
C Astral Shockwave (standard, recharges when the astral shade takes force damage) ? Force
ZZX +20 vs Reflex; 3d8 + 5 force damage and the target is pushed 5 squares. Close burst 3
C Radiant Chains (standard, recharge ? ? ) ? Radiant
ZZX +20 vs Fortitude; 3d8 + 5 radiant damage and the target is immobilized (save ends). Close burst 3
Alignment Unaligned Languages Common
Skills Stealth +20
Str 21 (+13) Dex 24 (+15) Wis 16 (+11)
Con 16 (+11) Int 18 (+12) Cha 21 (+13)

This is how the monsters come out after scraping from the raw text document.

When trying to parse a module with monsters in the above format it will give fatal error and stop module processing.

It does it with basically every dungeon magazine monster.

Insights or thoughts?

Primarch

[daumnation]

I have nothing of substance to add other than I hope this can be fixed because I use this heavily for FG2.:D

[primarch]

I have nothing of substance to add other than I hope this can be fixed because I use this heavily for FG2.:D

Hi!

Agreed! :D

Hopefully Eugenez will help us out. :)

Primarch

[Zeus]

@primarch: MM3 stat blocks are supported by the parser however the stat block you have posted looks like its in the original MM/DMG format which of course is supported too.

Can you post the error message you get from the Parser with this stat block?

As for Items, yes its the inclusion of the new Rare/Common tag that upsets the latest version of the Parser, I'm sure EugeneZ will address it in good time.

[primarch]

@primarch: MM3 stat blocks are supported by the parser however the stat block you have posted looks like its in the original MM/DMG format which of course is supported too.

Can you post the error message you get from the Parser with this stat block?

As for Items, yes its the inclusion of the new Rare/Common tag that upsets the latest version of the Parser, I'm sure EugeneZ will address it in good time.

Hi!

It was my recollection as well that the parser should be able to do it, so I did a more extensive test. This is what I found:

1. You are correct it does parse correctly. Problem was there is a handful of entries (which I will discuss below) that cause a fatal error. There were quite a few so I thought it was a more generalized problem, but once I persevered and removed each offending entry, the overwhelming majority parsed correctly.

2. What follows are the problem entries they all give the same result (so I posted that error only once), I have posted the raw data form off all problem entries. I tried to see why they are flawed but my inexperienced eye cannot find the issue.

10/21/2011 3:03:56 PM : Error data:
Eldritch Giants Level 18 Skirmisher
Large fey humanoid (giant) XP 2000
Initiative +12 Senses Perception +20; low-light vision
HP 171; Bloodied 85
AC 32; Fortitude 29, Reflex 31, Will 33
Resist 10 force
Saving Throws +5 against charm effects
Speed 8, teleport 6
m Eldritch Blade (standard, at-will) ? Force, Weapon
ZZX +21 vs Reflex; 2d6 + 7 force damage. Reach 2
r Force Missile (standard, at-will) ? Force
ZZX 20; +21 vs Reflex; 2d6 + 7 force damage.
C Sweeping Sword (standard, encounter) ? Force, Weapon
ZZX +21 vs AC; 3d6 + 7 force damage, and the target is knocked prone. Special: When charging, the eldritch giant can use this power in place of a melee basic attack.. Close blast 2
X Consume Magic (minor 1/round; at-will)
10/21/2011 3:03:56 PM : ERROR: Fatal Parse error on: Sweeping Sword (standard; encounter) ? Force, Weapon ; ZZX +21 vs AC; 3d6 + 7 force damage, and the target is knocked prone. Special: When charging, the eldritch giant can use this power in place of a melee basic attack.. Close blast 2
10/21/2011 3:03:55 PM : Start NPC-Stat Parse
10/21/2011 3:03:55 PM : NPC Missing Token Manifest
10/21/2011 3:03:55 PM : Start Familiar Parse
10/21/2011 3:03:55 PM : Start Vehicles Parse
10/21/2011 3:03:55 PM : Start Reference-Images Parse
10/21/2011 3:03:55 PM : Start Reference-Manual Parse
10/21/2011 3:03:55 PM : Start Ritual Parse
10/21/2011 3:03:55 PM : Start Skill Parse
10/21/2011 3:03:55 PM : Start Magic Item Parse
10/21/2011 3:03:55 PM : Start Condition Parse
10/21/2011 3:03:55 PM : Start Equipment Parse
10/21/2011 3:03:55 PM : Start Armor Parse
10/21/2011 3:03:55 PM : Start Weapon Parse
10/21/2011 3:03:55 PM : Start Power Parse
10/21/2011 3:03:55 PM : Reference parameters set
10/21/2011 3:03:55 PM : NPCs parameters set
10/21/2011 3:03:55 PM : Items parameters set
10/21/2011 3:03:55 PM : Adventure parameters set
10/21/2011 3:03:55 PM : Root parameters set
10/21/2011 3:03:55 PM : Subtree Verified
10/21/2011 3:03:55 PM : Top level verified
10/21/2011 3:03:55 PM : Starting parse

See post 2 for raw data of problematic entries.

Primarch

[primarch]

Post 2 raw txt data of problem entries.

Eldritch Giants Level 18 Skirmisher
Large fey humanoid (giant) XP 2000
Initiative +12 Senses Perception +20; low-light vision
HP 171; Bloodied 85
AC 32; Fortitude 29, Reflex 31, Will 33
Resist 10 force
Saving Throws +5 against charm effects
Speed 8, teleport 6
m Eldritch Blade (standard, at-will) ? Force, Weapon
ZZX +21 vs Reflex; 2d6 + 7 force damage. Reach 2
r Force Missile (standard, at-will) ? Force
ZZX 20; +21 vs Reflex; 2d6 + 7 force damage.
C Sweeping Sword (standard, encounter) ? Force, Weapon
ZZX +21 vs AC; 3d6 + 7 force damage, and the target is knocked prone. Special: When charging, the eldritch giant can use this power in place of a

melee basic attack.. Close blast 2
X Consume Magic (minor 1/round; at-will)
ZZX Targets an adjacent conjuration or zone created by an enemy; +21 vs. the Will of the creator of the conjuration or zone; the conjuration or

zone is destroyed, and its effects end. The eldritch giant’s attacks deal 3d6 extra force damage until the end of its next turn.
X Eldritch Field (minor; recharge ? ? ) ? Zone
ZZX Close burst 5; the blast creates a zone of rippling magic that lasts until the end of the encounter or until the eldritch giant uses this

power again. The zone is difficult terrain, and the giant’s attacks deal 1d6 extra force damage against creatures within the zone.
Alignment Evil Languages Elven, Giant
Skills Arcana +21, Insight +20
Str 21 (+14) Dex 12 (+10) Wis 22 (+15)
Con 19 (+13) Int 24 (+16) Cha 13 (+10)

Marauder Planestalkers Level 17 Lurker
Large aberrant magical beast XP 1600
Initiative +19 Senses Perception +13; low-light vision
HP 129; Bloodied 64
AC 31; Fortitude 28, Reflex 29, Will 28
Resist see planephase form; Vulnerable psychic; a marauder planestalker that takes psychic damage immediately ends planephase form.
Speed 7, teleport 5 see also planar evasion, planephase form
m Bite (standard, at-will)
ZZX +22 vs AC; 3d6 + 6 damage. Reach 2
M Body Snatch (standard; usable only in planephase form, encounter) ? Teleportation
ZZX +21 vs Fortitude; 4d6 + 6 damage, and both the target and the marauder planestalker are teleported to an extradimensional space that

completely encloses both creatures. The space is 2 squares high and 4 squares wide on each side. Other creatures do not have line of effect or

line of sight to the target or the planestalker, and the target and planestalker do not have line of sight or line of effect to other creatures.

If the planestalker takes 25 damage, if it is killed, or if the target is killed, the target and the planestalker return to the spaces they

occupied before being teleported, or to the nearest available spaces..
C Dimensional Disjunction (standard, encounter)
ZZX +19 vs Will; 2d6 + 3 damage, and the target takes 10 extra damage from melee attacks (save ends). Close blast 3
X Planar Evasion (immediate reaction, when the marauder planestalker is hit by a melee attack; recharge ? ? ? ) ? Teleportation
ZZX The marauder planestalker teleports 3 squares.
X Planephase Form (standard; sustain minor; at-will)
ZZX The marauder planestalker partially phases into another plane, gaining the insubstantial and phasing qualities until the end of its next turn

or until it attacks.
Alignment Unaligned Languages Deep Speech
Skills Bluff +14, Insight +18, Stealth +20
Str 16 (+11) Dex 24 (+15) Wis 21 (+13)
Con 21 (+13) Int 7 (+6) Cha 13 (+9)

Admiral Kada’ne, Githyanki Gish Overlord Level 19 Solo Skirmisher
Medium natural humanoid XP 12000
Initiative +17 Senses Perception +18
HP 724; Bloodied 362
AC 33; Fortitude 33, Reflex 31, Will 29
Saving Throws +5 (+7 against charm effects)
Speed 5, teleport 6
Action Points 2
m Silver Greatsword (standard, at-will) ? Psychic, Weapon
ZZX +24 vs AC; 1d10 + 6 damage plus 1d8 psychic damage.
M Blade Fury (standard, at-will) ? Psychic, Weapon
ZZX Kada’ne makes four silver greatsword attacks and can shift up to 2 squares after each. Effect: Kada’ne teleports 3 squares and gains a bonus

to his next attack roll equal to the number of creatures he hit with this attack
C Soulstorm Strike (standard, at-will) ? Lightning, Radiant
ZZX +22 vs Reflex; 2d8 + 4 lightning and radiant damage and ongoing 10 lightning and radiant damage. Close blast 5
A Astral Whirlwind (minor, recharge ? ? ) ? Psychic, Radiant
ZZX burst 1 within 10; +22 vs Fortitude; . the target is stunned until the end of its next turn.
R Telekinetic Leap (move, encounter)
ZZX 10; +22 vs or (whichever is lower); . Kada’ne or an ally within range can fly 5 square.
R Rescuing Banishment (immediate interrupt, when an ally in range is hit by an enemy attack, at-will) ? Teleportation
ZZX 10; targets the hit ally; +22 vs or (whichever is lower); . the target is banished to a demiplane and suffers no effect of the triggering

attack. It returns to a square within 10 squares of Kada’ne at the start of Kada’ne’s next turn
X Silver Banishment (free, when kada’ne hits a target with a weapon attack; recharges when first bloodied) ? Teleportation
ZZX The target is banished to a demiplane (save ends). It disappears from sight, cannot take actions, and cannot be targeted. On a save, the

target reappears in the space it last occupied. If that space is occupied, the target returns to the nearest unoccupied space of its choosing.
Alignment Evil Languages Common, Deep Speech, Draconic
Skills Insight +18, Intimidate +20
Str 26 (+17) Dex 22 (+15) Wis 18 (+13)
Con 21 (+14) Int 19 (+13) Cha 22 (+15)
Equipment: Silver Greatsword, Crystal Key, plate armor.

Ptolema, Medusa Archer Level 12 Artillery
Medium natural humanoid, female XP 700

HP 96; Bloodied 48 Initiative +12
AC 26, Fortitude 23, Reflex 25, Will 23 Perception+14
Speed 6
Immune petrification; Resist 10 poison


Traits
X The Rune of Poison’s Curse
ZZX If Ptolema moves more than 4 squares on her turn, she becomes weakened until the end of her next turn.

Standard Actions
m Serpent Hair (Poison) ? At-Will
ZZX Attack: Melee 1 (one creature); +17 vs. AC
Hit: 2d6 + 6 poison damage, and the target takes a -2 penalty to saving throws until the end of Ptolema’s next turn.
r Shortbow (Poison, Weapon) ? At-Will
ZZX Attack: Ranged 30 (one creature); +19 vs. AC
Hit: 1d10 + 7 damage, and the target takes ongoing 5 poison damage and is slowed (save ends both).

Triggered Actions
X Petrifying Stare ? At-Will
ZZX Trigger: An enemy starts its turn within 2 squares of Ptolema.
Effect (Opportunity Action): Close blast 2 (triggering enemy in the blast). The target is slowed (save ends).
First Failed Saving Throw: The target is immobilized instead of slowed (save ends).
Second Failed Saving Throw: The target is petrified until one of the following conditions is satisfied.
X * The use of an appropriate power, such as divine cleansing.
ZZX * Ptolema’s willing kiss (she might do this to gain information or to luxuriate in the victim’s fear before returning it to stone).
X * Ptolema is killed and her blood is applied to the stony lips of the victim before a full day passes.
X The Rune of Poison's Boon (Poison) ? Encounter
ZZX Trigger: Ptolema hits an enemy using shortbow.
Effect (No Action): The enemy takes ongoing 10 poison damage (save ends).
First Failed Saving Throw: The ongoing damage increases to 15.
Second Failed Saving Throw: The ongoing damage increases to 20.
Skills Bluff +15, Stealth +17
Str 14 (+8) Dex 22 (+12) Wis 17 (+9)
Con 18 (+10) Int 12 (+7) Cha 19 (+10)
Alignment Evil Languages Common
Equipment: shortbow.

Duergar Bloodmage Level 8 Controller
Medium natural humanoid XP 350

HP 87; Bloodied 43 Initiative +6
AC 22, Fortitude 19, Reflex 21, Will 20 Perception+6
Speed 5 Darkvision
Resist 5 fire, 5 poison


Standard Actions
m Quarterstaff (Weapon) ? At-Will
ZZX Attack: Melee 1 (one creature); +13 vs. AC
Hit: 2d8 + 7 damage.
r Bleeding Eyes (Psychic, Implement) ? At-Will
ZZX Attack: Ranged 5 (one creature); +11 vs. Will
Hit: 2d6 + 4 psychic damage, and the target is blinded until the end of the bloodmage’s next turn.
A Boiling Blood (Fire, Implement) ? Recharge ? ?
ZZX Attack: Area burst 2 within 10 (creatures in the burst); +11 vs. Fortitude
Hit: 2d6 + 4 fire damage, and the target takes ongoing 5 fire damage (save ends).

Minor Actions
R Infernal Quills (Poison) ? Encounter
ZZX Attack: Ranged 3 (one creature); +13 vs. AC
Hit: 1d8 + 3 damage, and the target takes ongoing 5 poison damage and a -2 penalty to attack rolls (save ends both).
X Blood Sacrifice ? At-Will
ZZX Effect: The bloodmage loses 5 hit points and chooses one of the following three effects.
X 1. The bloodmage gains a +4 bonus to his or her next attack roll.
ZZX 2. The bloodmage gains a +2 bonus to all defenses until the end of its next turn.
X 3. The bloodmage shifts up to its speed.

Triggered Actions
M Blazing Boils (Fire, Weapon) ? Encounter
ZZX Trigger: A creature hits the bloodmage with a melee attack.
Attack (Immediate Reaction): Melee 1 (the triggering creature); +13 vs. AC
Hit: 2d8 + 5 fire damage, and the target is immobilized (save ends).
Effect: The bloodmage shifts up to 2 squares.
Skills Arcana +14, Bluff +12, Dungeoneering +11
Str 13 (+5) Dex 14 (+6) Wis 14 (+6)
Con 15 (+6) Int 20 (+9) Cha 17 (+7)
Alignment Evil Languages Common, Deep Speech, Dwarven
Equipment: robes, quarterstaff.

See post 3 for remainder of entries.

Primarch

[primarch]

Post 3 remainder of problem entries.

Whirling Blades Automaton Level 6 Skirmisher
Small natural animate (blind, construct) XP 250
Initiative +9 Senses Perception +1; blindsight 10
HP 66; Bloodied 33
AC 20; Fortitude 18, Reflex 20, Will 16
Immune disease, knocked prone, poison
Speed 5
C Whirling Blades (standard, at-will)
ZZX +10 vs AC; 1d6 + 4 damage, and the automaton can shift 1 square for each target it hits up to 3 squares. Close burst 1
C Remove Mark (minor, at-will)
ZZX Close burst 5; targets an enemy marking the automaton; +9 vs Will; the mark ends.
C Bladed Destruction (when the automaton drops to 0 hit points)
ZZX +10 vs AC; 1d10 + 2 damage. The whirling blade automaton explodes, launching blades in all directions. Close burst 5
Alignment Unaligned Languages -
Str 14 (+5) Dex 19 (+7) Wis 6 (+1)
Con 10 (+3) Int 2 (-1) Cha 2 (-1)

Bugbear Wardancers Level 6 Skirmisher
Medium natural humanoid XP 250
Initiative +9 Senses Perception +5; low-light vision
HP 70; Bloodied 35
AC 20; Fortitude 18, Reflex 19, Will 18
Speed 7
m Flail Dance (standard, at-will) ? Weapon
ZZX +9 vs Fortitude; 2d6 + 4 damage, the target is pushed 2 squares and knocked prone, and the bugbear wardancer shifts 1 square..
C Flail Barrier (standard, recharge ? ? ) ? Weapon
ZZX Close burst 2; targets enemies; +9 vs or (whichever is lower); 5 damage, and the bugbear wardancer takes half damage from weapon attacks

until the end of its next turn.. no attack roll
C Flail Assault (standard, encounter) ? Weapon
ZZX +9 vs Reflex; 2d6 damage, and the target is pushed 2 squares and knocked prone. Close burst 2
X Combat Advantage
ZZX A bugbear wardancer deals 1d6 extra damage against any creature granting combat advantage to it.
Alignment Evil Languages Common, Goblin
Str 16 (+6) Dex 19 (+7) Wis 14 (+5)
Con 14 (+5) Int 11 (+3) Cha 16 (+6)
Equipment: heavy flail, hide armor.

Calmachia Level 9 Solo Controller
Large natural (dragon, living construct) XP 2000
Initiative +4 Senses Perception +5; darkvision
HP 400; Bloodied 200
AC 23; Fortitude 23, Reflex 20, Will 21
Saving Throws +5 (+7 against ongoing damage)
Speed 6, fly 8 (hover)
Action Points 2
m Bite (standard, at-will)
ZZX +15 vs AC; 2d8 + 6 damage, and the target is grabbed. Reach 2
m Claw (standard, at-will)
ZZX +15 vs AC; 1d10 + 6 damage, and the target slides 2 squares. Reach 2
M Double Attack (standard, at-will)
ZZX Calmachia makes two claw attacks
M Fling (standard, at-will)
ZZX Targets a creature Calmachia has grabbed; +13 vs Fortitude; 1d8+6 damage, and the target is pushed 6 squares and knocked prone. Miss: The

target is pushed 2 squares.
C Arcing Backlash (immediate reaction, when hit by an attack, at-will) ? Lightning
ZZX Close burst 10; +13 vs Reflex; 1d8 + 5 lightning damage, and the enemy is pushed 2 squares and knocked prone. targets only the attacker
C Breath Weapon (standard, recharge ? ? ? ) ? Force
ZZX +13 vs Reflex; 2d6 + 5 force damage, and the target is slowed (save ends) and pushed 3 squares. Close blast 5
C Bloodied Breath (free, when first bloodied, encounter) ? Force
ZZX Calmachia’s breath weapon recharges, and she uses it immediately.
C Commanding Voice (standard, encounter) ? Charm
ZZX Close burst 3; targets enemies; +11 vs Will; the target is dominated until the end of Calmachia’s next turn. Miss: The target is dazed until

the end of Calmachia’s next turn.
Alignment Unaligned Languages Common, Draconic
Skills Arcana +13, Intimidate +14
Str 23 (+10) Dex 10 (+4) Wis 12 (+5)
Con 20 (+9) Int 18 (+8) Cha 20 (+9)

Githyanki Warmongers Level 17 Soldier
Medium natural humanoid XP 1600
Initiative +15 Senses Perception +11
HP 162; Bloodied 81
AC 33; Fortitude 29, Reflex 31, Will 29
Saving Throws +2 against charm effects
Speed 6 see also telekinetic leap
m Silver Greatsword (standard, at-will) ? Psychic, Weapon
ZZX +24 vs AC; 2d10 damage plus 1d10 psychic damage, plus 3d8 extra psychic damage if the target is immobilized or stunned.
R Telekinetic Crush (standard, recharge ? ? )
ZZX 5; +22 vs Fortitude; 1d10 damage, and the target is immobilized (save ends). Medium or smaller target
R Telekinetic Leap (move, encounter)
ZZX Ranged 10; +22 vs or (whichever is lower); . the githyanki warrior or an ally within range can fly 5 squares.
M Soulsword Burst (standard; requires silver greatsword, encounter) ? Psychic, Weapon
ZZX +22 vs Will; 3d10 damage, and the target takes ongoing 5 psychic damage and is stunned (save ends both).. Close burst 1
Alignment Evil Languages Common, Deep Speech
Str 24 (+15) Dex 21 (+13) Wis 17 (+11)
Con 18 (+12) Int 16 (+11) Cha 21 (+13)
Equipment: Silver Greatsword, plate armor.

Astral Shade Level 17 Solo Lurker
Large immortal humanoid XP 8000
Initiative +19 Senses Perception +11; blindsight 10, darkvision
HP 448; Bloodied 224
AC 29; Fortitude 28, Reflex 30, Will 28
Immune disease, poison; Resist insubstantial; Vulnerable force (the next attack against the astral shade ignores its insubstantial)
Saving Throws +5
Speed fly 10 (hover); phasing
Action Points 2
m Astral Slam (standard, at-will) ? Radiant
ZZX +20 vs Fortitude; 3d8 + 4 radiant damage and the target gains vulnerable 10 to radiant damage until the end of the astral shade’s next turn.

If the astral shade is bloodied, the target is also restrained (save ends).
r Radiant Ray (minor, at-will) ? Radiant
ZZX One or two targets; +22 vs Reflex; 3d8 radiant damage, and the target slides 1 square. If the astral shade is bloodied, the attack instead

deals 3d8 + 5 radian damage and slides the target 2 squares.
M Wrathful Strike (immediate reaction, when damaged by an attack, at-will)
ZZX The astral shade teleports 4 squares and makes an astral slam attack or a radiant ray attack.
C Astral Shockwave (standard, recharges when the astral shade takes force damage) ? Force
ZZX +20 vs Reflex; 3d8 + 5 force damage and the target is pushed 5 squares. Close burst 3
C Radiant Chains (standard, recharge ? ? ) ? Radiant
ZZX +20 vs Fortitude; 3d8 + 5 radiant damage and the target is immobilized (save ends). Close burst 3
Alignment Unaligned Languages Common
Skills Stealth +20
Str 21 (+13) Dex 24 (+15) Wis 16 (+11)
Con 16 (+11) Int 18 (+12) Cha 21 (+13)

Where do you think the error is in these entries?

Thanks!

Primarch

[enygma]

I know it has been said already. Really looking forward to the parser update thanks again!

[Mortani]

Thanks so much for picking this up!

Versions beyond 4.0.118 seem to break the NPC Database functionality that works with the Module Workshop.

Also, had a few odd <h1 class="atwillpower"> in the Monster Vault scrape I just did.

[Dracones]

Anyone had any luck with parsing items from the compendium?

[zapoqx]

I haven't tried Dracones. However, considering that the old days of the items were not only sorted by doing +1 to +6 as separate entrees (as well as heroic/paragon/epic version of non-weapon/armor/neck items), but also there was the complete change on how items were displayed compared to before. Its probably requiring an entirely new code to scrape appropriately.

[EugeneZ]

As for Items, yes its the inclusion of the new Rare/Common tag that upsets the latest version of the Parser, I'm sure EugeneZ will address it in good time.

Weird, I thought I had this figured out... will address it ASAP. Thanks for the updates and reports, guys.

[lordsoth]

What is the status of the parser? Is there a version out there that has been updated and works well enough to scrape DDI stuff now?

I was trying to scrape mordenkainen's but it seems to crash on me.?

[EugeneZ]

What is the status of the parser? Is there a version out there that has been updated and works well enough to scrape DDI stuff now?

I was trying to scrape mordenkainen's but it seems to crash on me.?

The status is in the first post in this thread. The version out there has been updated to scrape MOST DDI stuff. The major exception is Items, and I'm working on getting that fixed ASAP. It sounds like there's a few other minor bugs with specific entries, but a lot of that is a game of cat-and-mouse where WotC changes the HTML of an entry to some non-standard format, seemingly at random.

[primarch]

Hi!

Any news on the update?

Thanks!

Primarch

[CScribe]

First of all thanks for the effort you have put into this very handy bit of software.

I am however having trouble...

When I attempt a Compendium Scrape of the Player's Handbook I get an error:

"The remote server returned an error: (500) Internal Server Error."

The last few lines displayed on the Processing Results window are:

14/12/2011 8:09:07 PM : Setting up Items
14/12/2011 8:09:07 PM : Epic Destinies scrape Complete!
14/12/2011 8:09:07 PM : Outputting powers to C:\Program Files\4EParser\Documents\Actual\Input\PHB\powers.t xt
14/12/2011 8:09:06 PM : Processing <EpicDestiny> - Deadly Trickster
14/12/2011 8:09:04 PM : Processing <EpicDestiny> - Eternal Seeker

Any tips or advice welcome.


P.S. I've included the stack trace:
--------------------------------
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Parse4E.Scrape.TopScrape(Int32 iType, Int32 iSourceID, String strDir, Boolean& booPowers, String txtEmail, String txtPassword, MetaList[]& MetaData, String[]& DynamicTokenList, Boolean booAppend, Boolean& booMonsters)
at Parse4E.Scrape.StartScrape(String strName, String strEmail, String strPass, String[]& DynamicTokenList, String& strDir, Boolean booScrapePower, Boolean booScrapeMonster, Boolean booScrapeFeat, Boolean booScrapeRace, Boolean booScrapeParagon, Boolean booScrapeEpic, Boolean booScrapeItem, Boolean booScrapeRitual, Boolean booScrapeTrap, Boolean booAppend, Boolean booScrapeMeta, Boolean booScrapeBackground, Boolean booScrapeFamiliar)
at Parse4E.frmPHBParse.ScrapeToolStripMenuItem_Click( Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStripItem.FireEventIntera ctive(EventArgs e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStripItem.FireEvent(Event Args e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
at System.Windows.Forms.ToolStrip.WndProc(Message& m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

[primarch]

Hi!

The issue is the rarity tags added. Thus the error when attempting to parse items.

Until the parser is updated the item parts of the scrape wont be accessible.

Hopefully well hear some news after the holidays.

Peter


First of all thanks for the effort you have put into this very handy bit of software.

I am however having trouble...

When I attempt a Compendium Scrape of the Player's Handbook I get an error:

"The remote server returned an error: (500) Internal Server Error."

The last few lines displayed on the Processing Results window are:

14/12/2011 8:09:07 PM : Setting up Items
14/12/2011 8:09:07 PM : Epic Destinies scrape Complete!
14/12/2011 8:09:07 PM : Outputting powers to C:\Program Files\4EParser\Documents\Actual\Input\PHB\powers.t xt
14/12/2011 8:09:06 PM : Processing <EpicDestiny> - Deadly Trickster
14/12/2011 8:09:04 PM : Processing <EpicDestiny> - Eternal Seeker

Any tips or advice welcome.


P.S. I've included the stack trace:
--------------------------------
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Parse4E.Scrape.TopScrape(Int32 iType, Int32 iSourceID, String strDir, Boolean& booPowers, String txtEmail, String txtPassword, MetaList[]& MetaData, String[]& DynamicTokenList, Boolean booAppend, Boolean& booMonsters)
at Parse4E.Scrape.StartScrape(String strName, String strEmail, String strPass, String[]& DynamicTokenList, String& strDir, Boolean booScrapePower, Boolean booScrapeMonster, Boolean booScrapeFeat, Boolean booScrapeRace, Boolean booScrapeParagon, Boolean booScrapeEpic, Boolean booScrapeItem, Boolean booScrapeRitual, Boolean booScrapeTrap, Boolean booAppend, Boolean booScrapeMeta, Boolean booScrapeBackground, Boolean booScrapeFamiliar)
at Parse4E.frmPHBParse.ScrapeToolStripMenuItem_Click( Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStripItem.FireEventIntera ctive(EventArgs e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStripItem.FireEvent(Event Args e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
at System.Windows.Forms.ToolStrip.WndProc(Message& m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

[EugeneZ]

Yeah, sorry guys, I have only had a spare hour here and there to work on this. I've got the issue nailed down and fixed, just need to do the legwork on testing and creating the binary. I'm on a brief vacation this weekend but I really hope to get this fix out in 2011, which gives me two weeks. Sounds doable.

[primarch]

Yeah, sorry guys, I have only had a spare hour here and there to work on this. I've got the issue nailed down and fixed, just need to do the legwork on testing and creating the binary. I'm on a brief vacation this weekend but I really hope to get this fix out in 2011, which gives me two weeks. Sounds doable.

Hi!

No worries EugeneZ, have a good time on your vacation. We'll be patient and await the good stuff. :D

Primarch

[Griogre]

I just re-parsed a bunch of modules and came across a bug I'd thought I'd mention that no one else has.

Apparently not all monsters have a space between the end of the Perception and the + or -. IE Balor is Perception+27 and not Perception +27 - while this doesn't crash the parser it does push the perception value down into special defenses.

[EugeneZ]

I noticed a problem with perception but didn't chase down the reason why. Thanks for that, I'll fix that in the next update. (Which is IMMINENT, I swear!)

[primarch]

Hi!

Any news on the update?

Thanks!

Primarch

[daumnation]

Getting the Items fixed is really important for this. This program is so crucial to my use of FG2 that Snmiteworks should try to aquire the code and include it as part of the program. However in the meantime, I'm hoping its not a done for.

[EugeneZ]

I thought I had the thing in the bag, but after running a few more test parses I noticed why the fix for items I put into my most recent build wasn't always working. Still a little work left to do. Coming soon, coming soon...

As for Smiteworks aquiring the code, that's up to Tenian, not me. However, a few things I'd like to point out:


Tenian's code is NOT written to be maintained by other developers. This thing is tens of thousands of lines of code across over a dozen files, all very function-driven. Long story short, I don't think ANYONE except for Tenian would want to seriously maintain this codebase. If Smiteworks was interested in obtaining this code, I think everyone involved would be better off if they just wrote their own parser from scratch (possibly using Tenian's code as a reference for edge cases -- but frankly I'm not sure it would be a good resource for that either).
The scraper component of the parser (which this thread indicates is its primary use-case) relies completely on the Compendium. With WotC starting playtests for DnD Next, there is the very serious question of whether the 4e content will even be available via the Compendium in the future.
Along the same lines, when DnD Next is released, even if WotC decides to keep the 4e Compendium around, demand for this scraper will drop. Solutions will be needed for Next content instead.


The reality is that this next patch to the parser that I will release will likely be the last. My interest in maintaining the code was entirely motivated by my own desire to fix the handful of issues it had/has for my own use. Once this patch is released, the scraper will work enough for my purposes (and hopefully the purposes of most of its users). Tenian has no desire to continue working on it (or play DnD in general, at the moment). So for any further work to continue on the parser, there would need to be someone out there interested in making changes (see my first point above for why this is unlikely) AND Tenian would need to be okay with them having his code. I believe both are extremely unlikely.

[Griogre]

EugeneZ brings up some good points about DDI. For those of you who are less familiar with development you should make sure you have the scrapped data files from each 4E resource you use to allow you to continue to play 4E as long as you wish. I believe a lot of people toss or reuse their scraped data files.

For those of you scraping this means you should really get and keep all the scrape data for each resource. This is not urgent but it will allow you to make any changes to the individual resource and parse it even if you can't get the data from DDI.

As 4E winds down there will be fewer and fewer errata updates which means the few updates will be easy to do by just changing a line or two in the scrapped data files - but only if you *do* have those data files.

[primarch]

Hi!

I constantly update my scrapped data files, so the point is well taken. At some point this year the files will assume their "last" form and those are the ones to get and archive.

My Thanks to Eugenez for working on the final update, since that one will permit us to get and archive the latest updates.

Thanks!

Primarch

[zapoqx]

Hmm...
But if I recall, it probably won't even be available until 2013 at the very least.

[Griogre]

Yeah, but ask the 3.5 guys how long those few online 3.5 Dragon and Dungeon Mags stayed up once 4E came out. And just how smooth did the transition from a downloaded character builder to an online one go? WotC is not exactly sterling in the way they transition from "old" to "new."

The other issue is that after EugeneZ releases the last update to the parser if DDI changes the database so something doesn't show up and if you don't have the old scrapes you are going to be out of luck. It might be good to give DrZuess' module decompiler a plug here.

[primarch]

Yeah, but ask the 3.5 guys how long those few online 3.5 Dragon and Dungeon Mags stayed up once 4E came out. And just how smooth did the transition from a downloaded character builder to an online one go? WotC is not exactly sterling in the way they transition from "old" to "new."

The other issue is that after EugeneZ releases the last update to the parser if DDI changes the database so something doesn't show up and if you don't have the old scrapes you are going to be out of luck. It might be good to give DrZuess' module decompiler a plug here.

Hi!

I think the important point is to download what you can now and save it and do so again once the updated parser is out to do the same.

Better to have most of the data, no matter how incomplete than to risk the data being pulled and not having had the opportunity to scrape it.

Primarch

[Bleak Midwinter]

Has anyone got this working on a Mac?

[primarch]

I thought I had the thing in the bag, but after running a few more test parses I noticed why the fix for items I put into my most recent build wasn't always working. Still a little work left to do. Coming soon, coming soon...

As for Smiteworks aquiring the code, that's up to Tenian, not me. However, a few things I'd like to point out:


Tenian's code is NOT written to be maintained by other developers. This thing is tens of thousands of lines of code across over a dozen files, all very function-driven. Long story short, I don't think ANYONE except for Tenian would want to seriously maintain this codebase. If Smiteworks was interested in obtaining this code, I think everyone involved would be better off if they just wrote their own parser from scratch (possibly using Tenian's code as a reference for edge cases -- but frankly I'm not sure it would be a good resource for that either).
The scraper component of the parser (which this thread indicates is its primary use-case) relies completely on the Compendium. With WotC starting playtests for DnD Next, there is the very serious question of whether the 4e content will even be available via the Compendium in the future.
Along the same lines, when DnD Next is released, even if WotC decides to keep the 4e Compendium around, demand for this scraper will drop. Solutions will be needed for Next content instead.


The reality is that this next patch to the parser that I will release will likely be the last. My interest in maintaining the code was entirely motivated by my own desire to fix the handful of issues it had/has for my own use. Once this patch is released, the scraper will work enough for my purposes (and hopefully the purposes of most of its users). Tenian has no desire to continue working on it (or play DnD in general, at the moment). So for any further work to continue on the parser, there would need to be someone out there interested in making changes (see my first point above for why this is unlikely) AND Tenian would need to be okay with them having his code. I believe both are extremely unlikely.

Hi!

Eugenez any news on the update?

Sorry for bugging ya, but I'm itching to get it and update my whole library. :)

Thanks for updating this, we all appreciate it!

Primarch

[EugeneZ]

I'm gonna take another run at the code tomorrow evening (Tuesday). I'll post an update afterwards.

[iowyn]

Hey all, sorry if this is covered, but I'm having trouble with a lot of traps, I assume it's only the ones with secondary triggers, but the only reference I saw about having:
Trigger 1
Secondary Trigger 2
Wasn't very clear, I messed around and couldn't get it right, an tips?

Also, is there any hope for an ETA on the items? It's fine to do that bit of work manually, but would be nice to have it parse over.

Thanks for your awesome and massive ongoing contributions EugeneZ. Hopefully for yourself you won't need to make another update and can have a breather (in that WotC don't mess with the format again).

[iowyn]

...and of course just read your last post. sorry, already had it in my head.

[iowyn]

Has anyone got this working on a Mac?
I'm sorry, I think because EugeneZ is stuck trying to handle it for PC, he's got enough work cut out for him.
It'd either be find a PC to run it on, or get one, because of the requirements from Net 4.0 to run and install the parser in the first place.

[EugeneZ]

I made some progress Tuesday but I still don't have a final solution nailed down that doesn't break anything. Hoping to take another swing at it tonight.

As for Mac compatibility, in theory, it might be possible to get the parser/scraper working in Mono. The biggest roadblock is the UI, but it's already pretty much just working from flatfiles (the save files), so it wouldn't be that hard to drop the GUI requirement. That said, there may be other incompatible libraries being used, I don't know. Maybe I'll look into it, but definitely not at any point in the near future.

[Bleak Midwinter]

So, I'm trying to use the parser to scrape from DDI with my login name and password set (at least, there's a nice green tick next to my login name).

However, I occasionally get told "Compendium returned a login screen. Account validation failed" (although it's obviously a working login). Worse yet, when it gets to scraping items from Heroes of the Fallen Lands, it simply crashes out. It managed to download a list of races and classes (names only). That's all that's in my 'input' folder.

Running on Windows 7.

[Griogre]

I'd check you login name and password. Remember for the DDI login the username is an e-mail address.

All scrapes will currently fail if you don't tell the scraper to not scrape items in books that have them. Scraping items will be on by default for HotFL. Double click on items and change the yes to no.

[Bleak Midwinter]

Right. It now parses (it was as you said) but for some reason, it lists Cleric Exploit powers as Rogue ones, fighter ones as Cleric ones, and so on. And I can't seem to manually edit after parsing. Yeesh.

[Griogre]

Yeah, that was a bug I told you about the power group headings are wrong but it does group things by top level class, while confusing - its purely cosmetic.

[Aloradianya]

Thank you so very much for working on this. I am in the process of scraping everything again as I lost it in a HDD crash. Trying to get set back up to reboot my campaign. I can't get the items part to work either, but this is something you already know about.

All of us appreciate the work you are doing.

[Bleak Midwinter]

Thanks, Griogre. Any idea what 'Split Feats in Bracketed Groups' means during parsing?

[Bleak Midwinter]

I've now parsed going on 20 things. Thank you so, so much EugeneZ for maintaining this and others for advice on how to use it!

Edit: has anyone tried parsing the Dark Sun Creature Catalogue? I've scraped the NPCs, but when I try and Parse it, it says that there's an error in line 15498 to do with <p>with Crystalline Web Attack.

Edit 2: have a similar issue with the Demonomicon. Error messages are along the lines of:
[quote]
the 'p' start tag on line 6481 position 763 does not match the end tag of 'il'./quote]

So I open up the .xml file in Notepad ++ and it looks okay to me.

Edit 3: errors were actually in the .txt file. Some of them are downloading <p="publishedIn"> and appending that randomly to some lines. Global replace got rid of it and it parsed.

[iowyn]

I've now parsed going on 20 things. Thank you so, so much EugeneZ for maintaining this and others for advice on how to use it!

Edit: has anyone tried parsing the Dark Sun Creature Catalogue? I've scraped the NPCs, but when I try and Parse it, it says that there's an error in line 15498 to do with <p>with Crystalline Web Attack. Opening up npcs.txt with Notepad++ it only goes up to line 5731, and searching for the text 'Crystalline' produces no hits. This leaves me somewhat bemused.

Edit 2: have a similar issue with the Demonomicon.


As far as I'm aware (and quite likely wrong), it's a line picked up during scraping that isn't required.
Open the file that has the problem, usually npc or trap scraps.
Search for the first < copy everything from <to>
Replace all lines in the file of <to> with nothing

That's an easy enough process in notepad.

[Bleak Midwinter]

Parsing "Martial Powers 2" it refers to a < (hex 0x3C) that I simply cannot find in the original .txt file. If I take the particular feat out of the text file (Longhand Bravo) it parses fine. The parser for some reason changes the feat name to match the Associated Powers. I've tried retyping it by hand into the powers.txt file in case there is a hidden character error but no joy.

Edit: the specific line that messes it up is:

<br />Associated Powers: Battle On, Brutal Setup, Deadly Returns, Great Dragon War Cry, Headstrong Bravery, Hold the Line<br />

[Griogre]

There's an extraneous html tag in powers. Find < in powers.txt to see what it is and then mass find and replace the tag (in your example its the <br /> ) with a space.

After scraping you should *not* have any html tags at all. Any you find - get rid of and replace with a space.

[Bleak Midwinter]

Thanks - it's very odd that it failed on just that one particular tag and no others.

[iowyn]

Another one I've got with Technique Feat Powers, is that it will parse fine, but when you check it in FG2 it will show the attack /with level/ but it doesn't show up under feat powers, and it doesn't include the full details. It only has the name, level, class and descriptive text.
The example from the PHB3 I'm working on trying to figure out is:


Dancing Cobra - Attack Technique Feat Power
You dodge and weave like a cobra, confounding your enemy and turning its attacks back against it.

At-Will ? Full Discipline, Implement, Psionic
Standard Action Melee touch
Target: One creature
Attack: Dexterity vs. Reflex
Hit: 1d10 + Dexterity modifier damage. If the target has made an opportunity attack against you during this turn, the target takes extra damage equal to your Wisdom modifier.
Level 21: 2d10 + Dexterity modifier damage.


Dancing Cobra Monk Attack 1
You dodge and weave like a cobra, confounding your enemy and turning its attacks back against it.

At-Will ? Full Discipline, Implement, Psionic

ZPOWER Dancing Cobra - Attack Technique
ZPOWER Dancing Cobra - Movement Technique

Dancing Cobra - Movement Technique Feat Power
You dodge and weave like a cobra, confounding your enemy and turning its attacks back against it.

At-Will ? Full Discipline, Psionic
Move Action Personal
Effect: You move your speed +2.


Any work done solving this before?

ah... it's an option in the 4E Parser... got it.

[Bleak Midwinter]

ah... it's an option in the 4E Parser... got it.

What was the option?


Unrelated, when I choose a book from the Compendium, is there a way to autofill it with everything except Items (other than loading from a config file which seems to be slower than just selecting manually). And the options that are pre-loaded for each book, do those always include everything that has detail in DDI, or are they just best guesses?

[iowyn]

What was the option?

Parse mode
Reference -> Powers/Feats/Features/Background -> Output Feat Powers to Library.

...I don't know why I hadn't even looked at these options.

[iowyn]

Is there anyway to scrap companions that are in the format of NPCs?
Like the Wolf Animal Companion in the Heroes of the Forgotten Kingdoms?... though I guess there are only a few of these.
Would they not already get picked up as an NPC in the format they are stored in the compendium?

[Griogre]

The format is too different for the HotFK animal companions compared to NPCs because the animal's stats are all relative to the owners stats and level - they change every level. You would be better off having the player create a second character and put the defenses and HP of the animal companion in the second character so the DM and easily drop the companion on the tracker.

For ease of play the player should put the companion's powers/attacks on the owning character's power list. That way he would never need to have the companion's mini or character sheet open except for maybe a skill check.

[iowyn]

Thanks Griogre, I had read that and a few other was around it. I guess there is far too much difference over the varieties of companions and familiars to make them part of FG2 themselves directly anyway.
I guess I'm lucky that my players aren't using them yet. Though I know it'll come up as they rotate through PC deaths. I think the 2nd pc sheet is the best solution so far.

[primarch]

I made some progress Tuesday but I still don't have a final solution nailed down that doesn't break anything. Hoping to take another swing at it tonight.

As for Mac compatibility, in theory, it might be possible to get the parser/scraper working in Mono. The biggest roadblock is the UI, but it's already pretty much just working from flatfiles (the save files), so it wouldn't be that hard to drop the GUI requirement. That said, there may be other incompatible libraries being used, I don't know. Maybe I'll look into it, but definitely not at any point in the near future.

Hi!

Any luck in nailing down a final fix?

Primarch

[EugeneZ]

I haven't had a chance to touch it since my last update. The place I work is launching a product next month and I've been swamped with preparations for that. Sorry... I really want this fixed and out of the way as badly as the rest of you, believe me. The earliest I can give it another shot is this weekend, if I have some spare time. I'll post an update Monday on whether that happens or not.

[primarch]

I haven't had a chance to touch it since my last update. The place I work is launching a product next month and I've been swamped with preparations for that. Sorry... I really want this fixed and out of the way as badly as the rest of you, believe me. The earliest I can give it another shot is this weekend, if I have some spare time. I'll post an update Monday on whether that happens or not.

Hi!

Thanks man, we are all extremely appreciative of your efforts! :)

Primarch

[tpkurilla]

I would like to add my appreciation as well.

I'm still getting crashes any time I try to get item data, and hope that this will finally overcome the issue.

-Thomas

[EugeneZ]

Zippo on the weekend, sorry. I'm starting to sound like a broken record. Um... I'm going to try and take an evening off work this Friday, just buckle down and knock this thing out. I'll post another update next Monday.

[iowyn]

at least after you've done this there shouldn't be a need for any more updates. it's unlikely/hopeful wotc won't change the format any more.

[DaffyRoth]

Just out of curiosity what version are we up to now?

[Pondscum09]

OK I have to say - how are people getting any of this to work? I always have a 500 error - no matter what I try to parse (and really all I want is the PHB).

What tips can people give me? What can I scrape manually (and how)?

[tpkurilla]

I've been having this issue since MME was introduced. I've usually been able to get scrapes to work if I set Items to 'No' when setting up the scrape.

There is supposed to be an update in the near future. Hopefully this issue will be addressed.

In the mean time, I've starting working on an alternative. Hopefully I'll be able to get it figured out before too much time passes.

-Thomas

[Griogre]

Like tpkurilla says set items to no when you scrape. Also you need to remove any tags from the text files. FYI make sure you manipulate the text files with Notepad - don't use a word processor. Try parsing and if you get an error, search on < if you find one mass find and replace the tag with a space. A html tag is everything from < to > including the < and >.

[Pondscum09]

Thanks, all. Getting there now.

The items are a pain, but I can probably work around those.

[iowyn]

Hey, EugeneZ how are you going with that last update. I know we're all looking forwards to throwing in the items.
I'm trying to figure out how I can make it work a lot easier with a random list for when I mention a bad guy with a weapon my PCs have decided they want to pry from them after a battle which I hadn't planned...

[EugeneZ]

The status is basically unchanged, which is why I haven't posted. I moved to a new apartment last weekend and have been busy unpacking and getting the new place set up. I also don't have an internet connection at the moment. I'm posting this from my mobile phone.

Once I get my internet and apartment situation sorted, I'll be in a better position to give estimates. I've been tinkering with the app for the last couple weeks, so hopefully we'll be able to put this app to rest at some point in April...

[iowyn]

Not to worry, I'm sure most of us know the grief of a move with arrival without a connection.
The fact we've got anyone who can even look at it is great.

We all look forwards to the next update.

Cheers

[primesuspect]

Just discovered this tool. Thank you for the awesome work!

[primesuspect]

After a full day of struggling, I cannot figure out how to get this tool to work. All I really need is the Monster Manual 4E exported to Fantasy Grounds.

The tutorial link on the wiki is broken (https://oberoten.dyndns.org/fgwiki/index.php/4EParser_Tutorial_1:_Your_First_Module), so I cannot find a guide on how to get this to work.

If someone could point me to a link on how to use this tool, I would really appreciate it. Guesswork has only led me to frustration.

[Griogre]

Here is this post that tell use how to do a module, though not a Monster Manual. https://www.fantasygrounds.com/forums/showpost.php?p=84732&postcount=8
The post is old so you should first get the parser and install it and then copy Eugenez beta exe over it. The items don't scrape since WotC combined them from multiple entries in DDI to one entry so you need to change items to no from yes when doing modules that contain magic items or the parser will crash.

Make sure you put use a unique Modestring for every module. Got to go. I'll be back later to answer any questions.

[primesuspect]

Thank you for the help so far. I have gotten that far by screwing around with it on my own, but no matter what I do, I get this:


4/8/2012 2:59:17 PM : ERROR:System.Xml.XmlException: 'Manual' is an unexpected token. The expected token is '='. Line 15, position 23.

<category name="4E Monster Manual" mergeid="" baseicon="2" decalicon="1">
</category>
</tables>
<effects>
</effects>
*** <reference4E Monster Manual static="true">
<weapon>
</weapon>
<armor>
</armor>
<equipment>

4/8/2012 2:59:17 PM : Checking XML
4/8/2012 2:59:17 PM : Output Completed
4/8/2012 2:59:17 PM : Copying Familiar Tokens
4/8/2012 2:59:17 PM : Copying Trap Tokens
4/8/2012 2:59:17 PM : Copying Monster Tokens
4/8/2012 2:59:17 PM : Outputing XML for Tokens
4/8/2012 2:59:17 PM : Outputing XML for Skill Challenge List
4/8/2012 2:59:17 PM : Outputing XML for Mount List
4/8/2012 2:59:17 PM : Outputing XML for Reference Manual Images
4/8/2012 2:59:17 PM : Outputing XML for Ritual/Alchemy List
4/8/2012 2:59:17 PM : Outputing XML for Magic Item List

[primesuspect]

Actually, the error changes; last time it was an error in the "languages" section under the "Otyugh".

I have a feeling it has to do with which files I'm pointing to in the selections. I feel like if I just plug in the correct files to the correct areas, the parse will work.

[Griogre]

Ah... one of those little details. It looks like you might have spaces in your modestring? You can't have any spaces in your modeString because the mode string is appended to XML elements and has to follow the rules for element names ie no spaces and alpha numeric characters only. Try using 4EMonsterManual or 4EMM1 as the modestring. Also just to double check you are using the beta exe, right?

Some of the data from DDI is not consistent. After you have scraped the information you need to search the text files and remove any HTML tags. IE search for < and then if you find one mass find and replace everything from < to > (including the < and >) and replace it with a space.

[primesuspect]

I am using the Beta, yes. Thanks!

[primarch]

Hi!

Any news on the update?

Thanks!

Primarch

[thargar]

First, great program. This has allowed myself and my friends to greatly expand and speed up our 4E playing experience on FG. Been just manually entering stuff into the individual modules to run campaigns. Major time saver, thanks.

Naturally, I found FGs after the change to DDI and I cant pull down the items either. I appreciate what EugeneZ has been able to do so far and thank him for it. I figure while I wait for the update, I might just mess around and see if I can get lucky and create a small equipment library of basic stuff using the parser.

Got a few questions based on my limited understanding of how the parser works.
1. Can the item text file be manually created just to get the simple items into FG. Mundane armor and equipment similar to what is in the 3.5 basic rule book that comes with FG? Should be easy enough to make the file in little time since it isnt that much information.

2. It appears that when the scrapper grabs the info, it puts it into a simple text file that the parser then uses to collect the information from. Does the scrapper put any specific code language into the text file? I didnt see anything specific in the text files that were created for powers, npcs, and the like.

I created a short item text file for some mundane armor and the parser is able to create an items file, but it just doesnt pull in any of the information. The client file does not include any item information. I used a similar format to the text files of the powers and npc files.

3. Is there a specific format the text file should follow or does the file have to have a specific name for the parser to see it? Simply called it armor.

Thanks for any info anyone can provide.

[Griogre]

Got a few questions based on my limited understanding of how the parser works.
1. Can the item text file be manually created just to get the simple items into FG. Mundane armor and equipment similar to what is in the 3.5 basic rule book that comes with FG? Should be easy enough to make the file in little time since it isnt that much information.

Yes. The parser works without the scraper. If you go Start -> All Programs -> 4E Parser -> Documents. You will pop a window with a Sample folder. Double click the Sample folder and you will see 3 more folders. Double click the Input one. That will bring up 3 more folders. Double click the PHB one. Inside you will find samples for armor, weapons and equipment. They have samples entries. I would suggest you copy and paste the files into the same folder with the scrapped files with which ever PH you are making. Then just follow the format of the items in the text files.



2. It appears that when the scrapper grabs the info, it puts it into a simple text file that the parser then uses to collect the information from. Does the scrapper put any specific code language into the text file? I didnt see anything specific in the text files that were created for powers, npcs, and the like.

I created a short item text file for some mundane armor and the parser is able to create an items file, but it just doesnt pull in any of the information. The client file does not include any item information. I used a similar format to the text files of the powers and npc files.

3. Is there a specific format the text file should follow or does the file have to have a specific name for the parser to see it? Simply called it armor.

Thanks for any info anyone can provide.

2-3) See the sample files at the location given above.

[thargar]

Thank you very much Griogre

[Griogre]

You're welcome. :)

[FSHSchmo]

Quick question on the parser. I am working on building my library but I have not been including my icons for info I scrape (ie the cover for the MM, or PHB). Is it possible to add this after the fact or do I need to stop now and start over?

[tpkurilla]

Quick question on the parser. I am working on building my library but I have not been including my icons for info I scrape (ie the cover for the MM, or PHB). Is it possible to add this after the fact or do I need to stop now and start over?

You don't need to re-scrape. Just load the "config" file for the parse (you *did* save the configuration, right?), then set the Thumbnail to the icon.

Then just re-parse it, and your icon will show up for the module in FG.

-Thomas

[Asturia]

When I try to parse items from the compendium I always get the following error message.
It doesn't matter from which sourcebook the items are coming.


See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Parse4E.Scrape.TopScrape(Int32 iType, Int32 iSourceID, String strDir, Boolean& booPowers, String txtEmail, String txtPassword, MetaList[]& MetaData, String[]& DynamicTokenList, Boolean booAppend, Boolean& booMonsters)
at Parse4E.Scrape.StartScrape(String strName, String strEmail, String strPass, String[]& DynamicTokenList, String& strDir, Boolean booScrapePower, Boolean booScrapeMonster, Boolean booScrapeFeat, Boolean booScrapeRace, Boolean booScrapeParagon, Boolean booScrapeEpic, Boolean booScrapeItem, Boolean booScrapeRitual, Boolean booScrapeTrap, Boolean booAppend, Boolean booScrapeMeta, Boolean booScrapeBackground, Boolean booScrapeFamiliar)
at Parse4E.frmPHBParse.ScrapeToolStripMenuItem_Click( Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStripItem.FireEventIntera ctive(EventArgs e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStripItem.FireEvent(Event Args e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
at System.Windows.Forms.ToolStrip.WndProc(Message& m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.261 (RTMGDR.030319-2600)
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
Parse4E
Assembly Version: 4.1.2.1
Win32 Version: 4.1.2.1
CodeBase: file:///C:/Program%20Files/4EParser/Parse4E.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.235 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.276 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.261 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.233 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.VisualBasic.Compatibility
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic.Compatibility/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.Compatibility.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.233 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Interop.Shell32
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Program%20Files/4EParser/Interop.Shell32.DLL
----------------------------------------
System.Web
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.272 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/System.Web/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Web.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

[FSHSchmo]

You don't need to re-scrape. Just load the "config" file for the parse (you *did* save the configuration, right?), then set the Thumbnail to the icon.

Then just re-parse it, and your icon will show up for the module in FG.

-Thomas

It appears I did not save my config for each scrape/parse... doh. Still have all my folders I scraped into just forgot to save... Good thing I wasnt to far, lol

[Zeus]

if all you want to do is add a thumbnail to a pre-built module that does not contain one already, quick tip:

Save your thumbnail image as thumbnail.png and then simply it add it to the root of the module file using WinZIP. i.e. Open the .mod file with WinZIP and then add your thumbnail.png to the archive.

[Griogre]

You have to have a DDI account to use the parser. If you do, make sure you are using your e-mail address and not your forum name for your login name.


When I try to parse items from the compendium I always get the following error message.
It doesn't matter from which sourcebook the items are coming.


See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Parse4E.Scrape.TopScrape(Int32 iType, Int32 iSourceID, String strDir, Boolean& booPowers, String txtEmail, String txtPassword, MetaList[]& MetaData, String[]& DynamicTokenList, Boolean booAppend, Boolean& booMonsters)
at Parse4E.Scrape.StartScrape(String strName, String strEmail, String strPass, String[]& DynamicTokenList, String& strDir, Boolean booScrapePower, Boolean booScrapeMonster, Boolean booScrapeFeat, Boolean booScrapeRace, Boolean booScrapeParagon, Boolean booScrapeEpic, Boolean booScrapeItem, Boolean booScrapeRitual, Boolean booScrapeTrap, Boolean booAppend, Boolean booScrapeMeta, Boolean booScrapeBackground, Boolean booScrapeFamiliar)
at Parse4E.frmPHBParse.ScrapeToolStripMenuItem_Click( Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStripItem.FireEventIntera ctive(EventArgs e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStripItem.FireEvent(Event Args e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
at System.Windows.Forms.ToolStrip.WndProc(Message& m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.261 (RTMGDR.030319-2600)
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
Parse4E
Assembly Version: 4.1.2.1
Win32 Version: 4.1.2.1
CodeBase: file:///C:/Program%20Files/4EParser/Parse4E.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.235 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.276 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.261 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.233 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.VisualBasic.Compatibility
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic.Compatibility/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.Compatibility.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.233 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Interop.Shell32
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Program%20Files/4EParser/Interop.Shell32.DLL
----------------------------------------
System.Web
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.272 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/System.Web/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Web.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

[FSHSchmo]

if all you want to do is add a thumbnail to a pre-built module that does not contain one already, quick tip:

Save your thumbnail image as thumbnail.png and then simply it add it to the root of the module file using WinZIP. i.e. Open the .mod file with WinZIP and then add your thumbnail.png to the archive.

Working like a charm!

Can I copy these modules from my laptop to my desktop and they work normally with my copy of FG2 there?

[Asturia]

You have to have a DDI account to use the parser. If you do, make sure you are using your e-mail address and not your forum name for your login name.
Yep, using an active account.
Everything goes fine except if I want to scrape any item.
I will try on another system to see if it happens also there.

[Zeus]

Working like a charm!

Can I copy these modules from my laptop to my desktop and they work normally with my copy of FG2 there?

Yes, simple copy the .mod files in the your FGII app data folder under the subfolder modules to the same location on the target system.

[Asturia]

Yep, using an active account.
Everything goes fine except if I want to scrape any item.
I will try on another system to see if it happens also there.
I've tried 2 systems one with windows xp and one with windows 7.
Both fully patched, and I just can't seem to scrape any items.
I can scrape anything else.

[Dakadin]

Items currently don't work when scraping. WotC changed the layout so the scraper can't process them.

EugeneZ was going to try to get one last fix for it but I think life has been keeping him busy with other things.

[Asturia]

Items currently don't work when scraping. WotC changed the layout so the scraper can't process them.

EugeneZ was going to try to get one last fix for it but I think life has been keeping him busy with other things.
Thanks for the info.
Guess I'll wait for the update.
I can add the items by hand.

[Griogre]

If you want to scrape now just set the Items setting to No and it will scrape everything else. You can then wait on the items. When EugeneZ gets it working then you can go back and only scrape the items.

[EugeneZ]

EugeneZ was going to try to get one last fix for it but I think life has been keeping him busy with other things.

That's exactly right, which is a shame because it's so close. I haven't had a moment to spare for DnD until now, hence this (lack of an) update.

I think it's safe to say that until I get some free time, the parser is what it is. I'm obviously terrible with dates, so "it's done when it's done."

[iowyn]

Good enough for me. But certainly looking forwards to the next update. Adding items would be fantastic.

[primarch]

Hi!

Eugenez, the item issue aside, while the parser ever manage themes?

Primarch1

[tpkurilla]

Hi!

Eugenez, the item issue aside, while the parser ever manage themes?

Primarch1

If I understood him correctly, he's working on getting items to work, and that will be his final update.

I'm hoping to be able to provide something useful as a replacement some time downline. Whenever I can get it to work, it will definitely include Themes.

-Thomas

[primarch]

If I understood him correctly, he's working on getting items to work, and that will be his final update.

I'm hoping to be able to provide something useful as a replacement some time downline. Whenever I can get it to work, it will definitely include Themes.

-Thomas

Hi!

Out of curiosity what platform are you using to write your version? I'm interested in learning to code in order to either help or do some on my own in the future and I like to see which options for coding are favored.

Thanks!

Primarch

[tpkurilla]

Hi!

Out of curiosity what platform are you using to write your version? I'm interested in learning to code in order to either help or do some on my own in the future and I like to see which options for coding are favored.

Thanks!

Primarch

I'm using C# and .NET. Getting the web pages into XML is trivial, and manipulating XML with .NET is also trivial.

My two main issues thus far have been logging in programmatically and, of course, finding time to dedicate to the project.

I plan on supporting the .monster exports from DDI as well. This will allow for conversion for all the modified/created monsters made with the Monster Builder.

I'm sure this is going to take me quite a bit of time to get into shape. But assuming that EugeneZ gets the items handled in the current parser, it'll definitely be enough to hold down the fort until I can manage it.

I will definitely let everyone know when I'm getting close.

[primarch]

I'm using C# and .NET. Getting the web pages into XML is trivial, and manipulating XML with .NET is also trivial.

My two main issues thus far have been logging in programmatically and, of course, finding time to dedicate to the project.

I plan on supporting the .monster exports from DDI as well. This will allow for conversion for all the modified/created monsters made with the Monster Builder.

I'm sure this is going to take me quite a bit of time to get into shape. But assuming that EugeneZ gets the items handled in the current parser, it'll definitely be enough to hold down the fort until I can manage it.

I will definitely let everyone know when I'm getting close.

Hi!

Looking forward to your work, your added functionality would be well appreciated indeed! :)

Primarch

[EugeneZ]

Hi!

Eugenez, the item issue aside, while the parser ever manage themes?

Primarch1
No, sorry, it will never manage themes unless Tenian decides to come back and code it.

I cannot stress how absolutely crazy the code for this app is.

The best idea for future support would definitely be to write something open-source from scratch and host it on git-hub or something. Hopefully tpkurilla will do just that. ;)

Until then, hopefully I can get this update squared away one of these days.

[Zeus]

I'm using C# and .NET. Getting the web pages into XML is trivial, and manipulating XML with .NET is also trivial.

My two main issues thus far have been logging in programmatically and, of course, finding time to dedicate to the project.

I plan on supporting the .monster exports from DDI as well. This will allow for conversion for all the modified/created monsters made with the Monster Builder.

I'm sure this is going to take me quite a bit of time to get into shape. But assuming that EugeneZ gets the items handled in the current parser, it'll definitely be enough to hold down the fort until I can manage it.

I will definitely let everyone know when I'm getting close.

:) I have just written a draft XSL stylesheet that handles DDI Adventure Tools .monster to FGII conversions. MM1/2 formats are supported at the moment but no MM3, that shouldn't be too much of an issue - just need some more time.

I am also working on a 4E Adventure Tools extension that will patch /importnpc functionality so that you can import the NPCs directly into your campaign. The upside to this approach is Adventure Tools storing its NPC data in a nice structured and uniform XML format, so conversion is very easy using XSL and XSLT, the downside is you can only export one monster at a time from Adventure Tools - so its not going to be a replacement for bulk module tools like the 4E Parser. Still for most DMs, I think it will prove a very useful method for getting NPCs into FGII with minimal hassle.

With regards to programmatically logging into DDI to extract the HTML data, PM me and I'll be glad to help you out. I have some Perl scripts which can already login and scrape the NPC data into FGII module format, which I can provide to give you a steer. Its pretty straightforward to extract the data on a module source basis - the trick comes in parsing the unstructured/non-uniform HTML that Wizards is using - enter the gazillion lines of conditional coding that Tenian has patched into the 4E Parser tool over the last few years and hence why poor old Eugenez is having such a hard time untangling the source.

Unfortunately as the standard for HTML is so poorly inconsistent across DDI Compendium entries, approaching this with something like XSL is not a trivial undertaking (believe me I know as I have tried), I fear the only way forward is going to be to write a line by line token based parser ... or petition Wizards to get the data into a structured state.

[Torgaard]

for most DMs, I think it will prove a very useful method for getting NPCs into FGII with minimal hassle.

For me, that's exactly right!

I kinda use the DDi Monster Builder for two things:

When I'm just kinda fishing for an interesting monster for an encounter. I'll just kinda prowl through the Builder, filtering by level range, looking for something fun. But I don't do this too terribly often.
But I frequently visit the Monster Builder once I've determined the foes my players will face. Sometimes WotC's critters are fine the way they are, but for me; all too often they feel like they could use another Power or two. If a monster looks like it's going to be using the same attack (or two) throughout an entire fight, I'm not happy. This is where the Monster Builder's "Add Power from a Monster" button is a godsend. I have way WAY too much fun playing with that Builder.


Lookin' forward to seeing another great tool added to the toolbox!

[primarch]

Hi!

Well it seems those with the skill have little time and those without skill (me) have the time. ;)

I've been picking the brains of several people and asked tpkurilla as well, so I ask Eugenez and DrZeuss, what program/skill set to learn would produce the highest yield to deal with Fantasy Grounds rule set development and modification?

Are there are sites/software, etc which you would recommend that would help me learning this?

I like putting my money where my mouth is, and its high time I took the plunge and learned how to do this and give something back to the FG community (like you guys).

Any input appreciated.

Primarch

[tpkurilla]

:) I have just written a draft XSL stylesheet that handles DDI Adventure Tools .monster to FGII conversions. MM1/2 formats are supported at the moment but no MM3, that shouldn't be too much of an issue - just need some more time.

I am also working on a 4E Adventure Tools extension that will patch /importnpc functionality so that you can import the NPCs directly into your campaign. The upside to this approach is Adventure Tools storing its NPC data in a nice structured and uniform XML format, so conversion is very easy using XSL and XSLT, the downside is you can only export one monster at a time from Adventure Tools - so its not going to be a replacement for bulk module tools like the 4E Parser. Still for most DMs, I think it will prove a very useful method for getting NPCs into FGII with minimal hassle.

With regards to programmatically logging into DDI to extract the HTML data, PM me and I'll be glad to help you out. I have some Perl scripts which can already login and scrape the NPC data into FGII module format, which I can provide to give you a steer. Its pretty straightforward to extract the data on a module source basis - the trick comes in parsing the unstructured/non-uniform HTML that Wizards is using - enter the gazillion lines of conditional coding that Tenian has patched into the 4E Parser tool over the last few years and hence why poor old Eugenez is having such a hard time untangling the source.

Unfortunately as the standard for HTML is so poorly inconsistent across DDI Compendium entries, approaching this with something like XSL is not a trivial undertaking (believe me I know as I have tried), I fear the only way forward is going to be to write a line by line token based parser ... or petition Wizards to get the data into a structured state.

I use a tool called HtmlAgility (with a few of my modifications) to take the raw HTML spewed by the compendium, and render it into super-friendly XML. From there, manipulation becomes substantially easier.

I've used the HeroLab open source C++ code (with my own modifications) to scrape every category in the DDI. So I can personally get all of the data...backgrounds, poisons, diseases, themes, glossary...everything. But the code is in no state for the average user to use. The monolithic nature of this code is something that I very much want to address with my own implementation.

I am hoping to get scraping code integrated with the rest of my C# code base to allow a complete tool to be packaged for general use. The level of manipulation possible with this data in XML could conservatively be called awesome!

To be quite honest, I am a rank amateur at dealing with the http session states and interaction required to get this done. I'm am absolutely certain that what I need done is doable with the C# libraries available. I just have not yet acquired the necessary expertise to manage it.

In the mean time, I press on with what little time I can spare tackling the parsing of the available data formats. Even with the scraped data jammed into XML, there is still enough "free-form" formatting to complicate the extraction of the pertinent data.

Once I get the data extracted into a well-constructed XML document, however, a whole host of operations become quite trivial.

My plan is to get this toolbox put together using open source code, and the make it available for all to use. To that end, I would definitely appreciate any help with getting a suitable scraper implemented.

Any code that I can see, no matter the language, that actually manages the DDI login and state maintenance would be helpful. I firmly believe that I'll get to a point where all that data will converge in my brain, and I'll finally "get it".

I've been purposely low-key about what I'm doing because I know that this is going to take some calendar time to get done. I really wish that I could move along with this more quickly, but life does have its demands that take precedence. I have had a few bursts of high productivity in between my other commitments. If I can just get a few more, I could start to show something useful from my efforts.

I will definitely let everyone know when I have something useful.

-Thomas

[Spikenog]

Wondering if the parser has been fixed to work with items. Any help would be greatly appreciated.

Scraping somethings this afternoons and the parser fails to continue once it reaches items. I receive the following:
-----------------------------------------------------------------------------

See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Parse4E.Scrape.TopScrape(Int32 iType, Int32 iSourceID, String strDir, Boolean& booPowers, String txtEmail, String txtPassword, MetaList[]& MetaData, String[]& DynamicTokenList, Boolean booAppend, Boolean& booMonsters)
at Parse4E.Scrape.StartScrape(String strName, String strEmail, String strPass, String[]& DynamicTokenList, String& strDir, Boolean booScrapePower, Boolean booScrapeMonster, Boolean booScrapeFeat, Boolean booScrapeRace, Boolean booScrapeParagon, Boolean booScrapeEpic, Boolean booScrapeItem, Boolean booScrapeRitual, Boolean booScrapeTrap, Boolean booAppend, Boolean booScrapeMeta, Boolean booScrapeBackground, Boolean booScrapeFamiliar)
at Parse4E.frmPHBParse.ScrapeToolStripMenuItem_Click( Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStripItem.FireEventIntera ctive(EventArgs e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStripItem.FireEvent(Event Args e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
at System.Windows.Forms.ToolStrip.WndProc(Message& m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.269 (RTMGDR.030319-2600)
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
Parse4E
Assembly Version: 4.1.2.1
Win32 Version: 4.1.2.1
CodeBase: file:///C:/Program%20Files%20(x86)/4EParser/Parse4E.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.278 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.282 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.269 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.233 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
Microsoft.VisualBasic.Compatibility
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic.Compatibility/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.Compatibility.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.233 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Interop.Shell32
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Program%20Files%20(x86)/4EParser/Interop.Shell32.DLL
----------------------------------------
System.Web
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.272 built by: RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/System.Web/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Web.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
-----------------------------------------------------------------------------

[Griogre]

The parser still doesn't work with items. Eugenez is working on a fix but real life and lack of time have not allowed him to release a fix. You may parse anything but items by telling the scraper not to scrape items by changing items from yes to no after you have selected a book.

[EugeneZ]

So I tried to take a crack at this today but I'm now getting a different error than before (by before I mean months ago when I first released my patched binary).

I tried filtering items by source in Firefox/Chrome/Opera/Internet Explorer, all to no avail. I get a 500 server error. Am I doing something wrong or is the Compendium simply broken? If it's broken, I'm not sure there's anything I can do with the scraper that'll fix that... Maybe I can try changing the way it filters items (the global listing of items appears to work), but that will be a large undertaking because right now the code depends heavily on filtering by Source to limit the amount of work it needs to do.

Can someone else confirm that filtering items by source is not working? Or explain what I'm doing wrong?

In other news, to aid in the creation of alternate scraping tools, I've open sourced the login code I wrote. I cleaned it up a bit. Well, more than a bit. And converted it to C#. I can't open source anything else because Tenian doesn't want his code in the open, but since I wrote the login code for him years ago when he first made the scraper, I figure it's fair game.

https://github.com/EugeneZ/compendium

This code will allow you to programatically log into the Compendium. The scraping is up to you...

[Zeus]

EugeneZ - I passed a Perl script I wrote to another community member here on the forums (can't remember their name now) who was attempting to write another parser. The Perl script demonstrates how you can programatically access DDI by source and filter type to scrape content - I even included an XSL stylesheet to convert the HTML data to XML for FGII. I'll dig it out and post up a copy here, if anyone else wants to have a go.

DDI's Item API changed a while back I believe (can't remember the extract date). Anyhow, you can now longer search on Items and filter by Source alone; you need to also include the Item sub category e.g. armor, equipment, item, and all the magic item types like rings.

Searching Items on source alone results in an empty results set when using the official web front end. Adding the category type works though.

[Griogre]

Thanks for releasing the Login code EugeneZ. I've gone looking for the API but I've never been able to find the up to date one.

[tpkurilla]

Thanks for posting what you could EugeneZ!

I'm downloading it now. Hopefully I'll be able to spend some time on it this weekend.

-Thomas

[Zeus]

Did some poking around in the Javascript and came across the updated filter strings used with KeywordSearchWithFilters method when searching for Items.

I can now get items by source (using Adventurer's Vault as an example) using the following url format:


https://https://www.wizards.com/dndinsider/compendium/CompendiumSearch.asmx?op=KeywordSearchWithFilters? keywords=&nameOnly=false&tab=Item&Filters="null|-1|-1|-1|-1|-1|-1|null|10|null"

This returns and XML data set containing IDs of all items in Adventrurers Vault. The new filter string format is


Category Name (null or string)
Item Level (low) (-1 or positive integer)
Item Level (high) (-1 or positive integer)
Item Cost (low) (-1 or positive integer)
Item Cost (high) (-1 or positive integer)
Item Enhancement (low) (-1 or positive integer)
Item Enhancement (high) (-1 or positive integer)
Item Rarity (null or positive integer)
Item Source ID (null or positive integer)
Item Category ID (null or positive integer)


Hope that helps.

EDIT: It would appear that when searching on all items from a Source Book where the sourcebook contains a lot of items; the generation of the returned dataset can timeout, especially when DDI servers are under load. This may also explain the historic hit/miss scenarios when trying to scan for DDI items by source only (i.e. server 500 or timeout errors). I am adapting my perl script to search by source and item type so as to reduce the dataset size. It means having to do more searches (one for each item type, total of about 24 but I can't see any other way around this at the moment.

[EugeneZ]

Thanks for the sleuth work, DrZeuss. I'll give that approach a shot.

[EugeneZ]

Okay, quickly threw this together. If it works, all credit goes to DrZeuss who figured out the correct filter string after it took me months to figure out what the problem with the parser even was.

If it doesn't work, I think a few tweaks should put it in order and I have more time these days.

Let me know how it goes:

https://www.eugenez.net/downloads/parser/4.1.2.3/Parse4E.exe

Remember, that's just the binary. Overwrite the one in the installation folder from Tenian's latest installer (I don't have Visual Studio Pro, so I can't use his installer).

[Griogre]

I tried to run AV and got the following error message:
Compendium returned a login screen. Account validation failed.

This happened right after Heavy Barding for Huge Creatures.

https://i105.photobucket.com/albums/m222/Griogre/4EScrapererror.png

After I pressed the OK button the scraper continued with rituals and completed.

It was nice to see the master work and barding scraping again. The armor.txt and mi.txt files were not actually updated.

[EugeneZ]

I tried to run AV and got the following error message:
Compendium returned a login screen. Account validation failed.

Hm. I'll take a look tonight. Thanks for testing it out.

[Griogre]

You're welcome, glad to help. I had some time so I ran Mordenkainen's Magnificent Emporium as well. The items didn't scrape into mi.txt other than one: Shield of Prator Level 1

The non magical armor and weapons scraped fine as did the feats. Equipment did scrape but prices for big ticket items like buildings are clipped after the first comma. Thus a castle's price is displayed as 250 when the price should be 250,000 and a Castle with dungeon has a displayed price of 1 instead of 1,000,000. For some reason Traverler's Kit is showing up under Buildings instead of with Gear and it has a colon on the line above.

The NPCs are failing to parse as well. They are all scraped, but don't parse correctly, probably due to them being Companions and having Healing Surges and Surge Values in the NPC headers. Only the first one, Anaxana, parsed and the 4E NPC sheet had problems with her as well - giving the error Unable to Migrate read-only NPC: Anaxana in the chat window.

[Zeus]

@Eugenez - your welcome. Glad to help. I would suggest pulling down the search.js javascript file from DDI. Within the script you will find definitions of the filter strings used with the KeywordSearchWithFilters method. This has been my primary source for developing my Perl based scraper.

I managed to get my Perl script updated so that it can scrape Items sub-divided by item type - this to mitigate against server timeout errors. I have tested the new version against a good broad selection of rulebooks including AV1 and 2 and MME and it appears to be working with good stability. All known content including newer types like Themes and Classes are correctly pulled down.

One thing I have noticed is that DDI's indexes are stale; e.g. If you search for all Glossary entries for the DMG using KeywordSearchWithFilters, the last entry in the returned XML dataset points you to the Glossary Entry for the Bluff Skill with a Glossary Item ID of 3. Consequently searching for Glossary Item 3 with the display method (to retrieve the HTML for the entry) results in a server 500 error as it does not exist. There are quite a few such examples littering the database.

At the moment my Perl based scraper tool logs you into DDI, presents a list of rulebooks/soucebooks; once a selection is made it then generates a folder for the selected module and creates subfolders (for each content type npc, item, etc. etc) and downloads the raw HTML pages (minus the embeded scripts and stylesheet data) for all the associated entries. I am writing a separate tool (html2parser) which uses XSLT to transform the HTML source data into 4EParser compatible syntax ready for Parsing. So far I have NPCs working (although I noticed all the old formatted creatures have been reformatted to follow the new MM3 format) and I am planning to work on Items next this w/e. I have taken the split program approach as I prefer to pipeline my module production work - i.e. scrape/convert/parse in parallel to help with cutting production time down and improve productivity. Its also useful as it allows me to use the data in a variety of ways - private off-line database :)

I am still not convinced I will be able to get a pure XSLT based solution working 100% in all cases and so I may have to fall back to using a line by line regex based approach; time will tell.

[DaffyRoth]

Hello! Alrighty I am using Parser ver 4.1.2.3, and every time I go to scrape the compendium I get an Unhandled exception. It says:

************* Exception Text **************
System.IO.FileNotFoundException: Could not load file or assembly 'System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
File name: 'System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
at Parse4E.Scrape.Compendium_Login(String strUser, String strPass)
at Parse4E.Scrape.TopScrape(Int32 iType, Int32 iSourceID, String strDir, Boolean& booPowers, String txtEmail, String txtPassword, MetaList[]& MetaData, String[]& DynamicTokenList, Boolean booAppend, Boolean& booMonsters)
at Parse4E.Scrape.StartScrape(String strName, String strEmail, String strPass, String[]& DynamicTokenList, String& strDir, Boolean booScrapePower, Boolean booScrapeMonster, Boolean booScrapeFeat, Boolean booScrapeRace, Boolean booScrapeParagon, Boolean booScrapeEpic, Boolean booScrapeItem, Boolean booScrapeRitual, Boolean booScrapeTrap, Boolean booAppend, Boolean booScrapeMeta, Boolean booScrapeBackground, Boolean booScrapeFamiliar)
at Parse4E.frmPHBParse.ScrapeToolStripMenuItem_Click( Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStripItem.FireEventIntera ctive(EventArgs e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStripItem.FireEvent(Event Args e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
at System.Windows.Forms.ToolStrip.WndProc(Message& m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

It happens when the log get to logging in to the compendium, however my username and pass are correct so I dont know what could be casuing it. I have been out of the game for awhile so I am kind rusty with all this again. So any help would be appreciated.

[Griogre]

Are you using your Username instead of your e-mail address? You need to use your e-mail address.

[DaffyRoth]

Nope using my email address.

[Griogre]

Hmm. Login would have been the easy answer. I'll take one last stab another easy possibility: Are you logged into DDI with your browser while running the Parser? If so try closing the browser. Also to double check, you did install the parser and then copy the lastest exe in the first post of this thread over the exe in program files, right?

If its not that, then you should probably install the lastest .NET framework. You might find this post helpful: https://social.msdn.microsoft.com/Forums/en-US/netfxsetup/thread/e6cd11ce-210e-4ed1-ae4c-5e251246797a/

[primarch]

Hi!

I have a similar problem, but with some odd quirks.

After downloading the exe and replacing the older one in the program folder I scraped the MMM book successfully, however only one item "Shield of Prator Level 1" was present on the item.txt file.

I tried with other books and get the error when it reaches items:

************** Exception Text **************
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Parse4E.Scrape.TopScrape(Int32 iType, Int32 iSourceID, String strDir, Boolean& booPowers, String txtEmail, String txtPassword, MetaList[]& MetaData, String[]& DynamicTokenList, Boolean booAppend, Boolean& booMonsters)
at Parse4E.Scrape.StartScrape(String strName, String strEmail, String strPass, String[]& DynamicTokenList, String& strDir, Boolean booScrapePower, Boolean booScrapeMonster, Boolean booScrapeFeat, Boolean booScrapeRace, Boolean booScrapeParagon, Boolean booScrapeEpic, Boolean booScrapeItem, Boolean booScrapeRitual, Boolean booScrapeTrap, Boolean booAppend, Boolean booScrapeMeta, Boolean booScrapeBackground, Boolean booScrapeFamiliar)
at Parse4E.frmPHBParse.ScrapeToolStripMenuItem_Click( Object sender, EventArgs e)
at System.Windows.Forms.ToolStripItem.RaiseEvent(Obje ct key, EventArgs e)
at System.Windows.Forms.ToolStripMenuItem.OnClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleClick(Eve ntArgs e)
at System.Windows.Forms.ToolStripItem.HandleMouseUp(M ouseEventArgs e)
at System.Windows.Forms.ToolStripItem.FireEventIntera ctive(EventArgs e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStripItem.FireEvent(Event Args e, ToolStripItemEventType met)
at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEven tArgs mea)
at System.Windows.Forms.ToolStripDropDown.OnMouseUp(M ouseEventArgs mea)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ScrollableControl.WndProc(Mes sage& m)
at System.Windows.Forms.ToolStrip.WndProc(Message& m)
at System.Windows.Forms.ToolStripDropDown.WndProc(Mes sage& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
Parse4E
Assembly Version: 4.1.2.3
Win32 Version: 4.1.2.3
CodeBase: file:///C:/Program%20Files%20(x86)/4EParser/Parse4E.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
Microsoft.VisualBasic
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
----------------------------------------
Microsoft.VisualBasic.Compatibility
Assembly Version: 10.0.0.0
Win32 Version: 10.0.30319.1
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic.Compatibility/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.Compatibility.dll
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
Interop.Shell32
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Program%20Files%20(x86)/4EParser/Interop.Shell32.DLL
----------------------------------------
System.Web
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_32/System.Web/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Web.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

I've tried installation .net framework 4.5. same error
Removed 4.5 and reinstalled 4.0. same error
Removed and reinstalled the parser and exe. same error
Turned off firewall. same error

I've used my e-mail as user name and correct password.

At no moment was I logged into DDi via browser.

I'm gonna try on a different computer and report back my findings.

Primarch

[Zeus]

EDIT: It would appear that when searching on all items from a Source Book where the sourcebook contains a lot of items; the generation of the returned dataset can timeout, especially when DDI servers are under load. This may also explain the historic hit/miss scenarios when trying to scan for DDI items by source only (i.e. server 500 or timeout errors). I am adapting my perl script to search by source and item type so as to reduce the dataset size. It means having to do more searches (one for each item type, total of about 24 but I can't see any other way around this at the moment.

From the looks of the error message, the 4EParser is falling over due to an Internal Server 500 Error being returned from the DDI web servers. (see my previous comment above).

I just tested my Perl script and it pulls down Items OK but only when I search by sourcebook and filter on item types.

[primarch]

From the looks of the error message, the 4EParser is falling over due to an Internal Server 500 Error being returned from the DDI web servers. (see my previous comment above).

I just tested my Perl script and it pulls down Items OK but only when I search by sourcebook and filter on item types.

Hi!

Pardon my ignorance, but does this mean it does not scrape becuase the server is too busy?

I've tried scraping selecting one book and just "items" but I get the same error.

Thanks!

Primarch

[Zeus]

In straightforward terms it means if the 4EParser is attempting to identify all items in a sourcebook using a single request, its likely to get server 500 errors, if the dataset returned is very large. To mitigate, the 4EParser will need to be modified so that it searches by sourcebook and item type (e.g. Rings) for items.

In addition, I mentioned in another post, that I have encountered a lot of stale data in the indexes for DDI i.e. pointers to data records that do not exist. If your getting hit/miss results with some sourcebooks but not with others (e.g. PHB which I have tested with my Perl script and I know works) then it could also be the stale indexes that are causing the problem.

To further explain let me outline how DDI works:

1) Wizards have exposed a web method called GetFilterSelect() which interrogates the DDI database and returns an XML data set containing all the DDI Compendium filter options and more importantly their IDs.
2) You can then use another web method Wizards have exposed called KeywordSearchWithFilters() which when used in conjunction with a filter string containing your search filter criteria and an ID from step 1, returns another XML data set, this time containing the IDs of all matching content in the database.
3) Finally you can use another method Wizards have exposed called Display(), which when used in conjunction with the content IDs returned from KeywordSearchWithFilters(), will return an HTML page containing the content.

So, if the data returned in step 2 is invalid in anyway, when the parser tries to execute Display() (or equivalent) to reference the content HTML page it will fail with an internal server 500 error.

To overcome this problem, the 4EParser would have to be modified to catch and maange this type of server exception gracefully. Problem is how do you know the data doesn't exist until you try to reference it?

[Griogre]

primarch, what this means to you right now is the 4E Parser doesn't scrape items, it hasn't been able to for about a year now. You can tell it to not scrape items by setting items to no. I'm not sure if Eugenez is going to take another stab at fixing this or not the 4E Parser isn't his program, and he seems to be short on time. On the bright side, with the latest fixes you can actually scrape mundane equipment again.

[primarch]

Hi!

Thank you for the explanation guys!

I just thought I was doing something wrong and it was possible to scrape items if done correctly.

No biggie, I got all the books done with the old format. Although it would be nice to get the MME book scraped, I've manually entered the ones I've needed in my games so far.

Looks like the old parser is very difficult to work with/patch, hopefully tpkurilla will have luck making it from the ground up. Perhaps in the end that will be the easiest approach.

Dr. Zeuss, go back to fiddling with your 4e enhanced images. I'm thinking we could all use condition/effects a whole lot more than updating the items. :)

Thanks for all the great stuff you guys do!!

Primarch

[DaffyRoth]

Heh ya I had it opened in another window but I just closed it and still throws the same error. I am going to try and update the framework and report back.

Oh I noticed that a lot of my damage and perception scores are off since the FG update. The last I used was 2.8.1. Was wondering if there was a new format to parse with?

Heh, ya I installed the frame work and still same thing. I have not re-installed this in like forever and I dont even know where the installer is anymore. I do have the latest .exe though.

[Griogre]

In one of the recent iterations of the FG character sheet the passive perception and passive insight listings were removed from the first "Main" tab and an active perception roller added. Otherwise your damage and perception should be the same. You might want to check that your stat mods are right.

You might have an older version of the parser, the last version released by Tenian was 4.0.118 and the next exe was built to sit on top of that. The first post of this thread has a link to the last installer for the parser by Tenian and the latest exe by Eugenez. There is also a link to the 4.0 .NET framework which is the one you want.

[DaffyRoth]

Ah so what the problem was, as far as I could tell, was that I need to log out of my DDI account. So all is working as intended.

The issue with perception is in the Monster Manual monsters. I did all my own mods for them and I doubled checked them to make sure they worked properly and everything was in the right place. But now post update the perception score reads 0 but below it in italics it says perception then the score. So I was wondering if there was a new outline for the monsters.

[Griogre]

The issue is DDI is not consistent in how the perception mod is input into their database. To fix this after you scrape a something that has NPCs, go into the npc.txt file and mass change Perception+ to Perception + and Perception- to Perception - and that will fix your problem and then re-run the parser using the updated text file.

By the way, I'd suggest you keep all the scraped text files for each library module so if you need to make a fix you don't have to re-scrape.

[daumnation]

Is it me or are items still stalling out the parser, the latest one, 4.1.2.3?
I was under the impression this issue had finally been resolved. But when I went to scrape the parser got the the Error 500 unhandled exception error and stalled there.
Any ideas or was I incorrect in thinking the items could be scraped?

[Griogre]

Magic items still don't scrape. However mundane items will scrape now if you don't get a 505. The 505 is basically a server timeout error which you will get if the WotC servers are under load when you parse. Thus the only reward you get for trying to scrape items is getting the mundane items out of one of the PHs or AV item types. You will have to sit through it going through the magic items to no avail get the mundane items, though.

[valeros]

New item parser here (not Tenian's): https://www.fantasygrounds.com/forums/showthread.php?t=17485

[nyhotep]

I had the same error as Joseph above.

Switching to xp compatibility seems to have solved it (touch wood)

[judesudbury]

Hi everybody. New to FG and new to the 4e Parser. I've read through much of this thread. I've installed the original parser and I've downloaded the updated exe file. I'm attempting to parse powers from the Dungeon Survival Handbook, and it scrapes fine, but when I attempt to parse into a module, I keep getting:

ERROR: line:Effect: You gain a +2 power bonus to all defenses until the end of your next turn.

If I edit the text file and remove the whole power associated with that line, I just get:

ERROR: line:

Would anybody be able to help me?

Many thanks,
Jude

[nyhotep]

I'm having a problem parsing Monster Manual 1;

The last lines are
12/12/2012 12:31:11 : ERROR: line:
12/12/2012 12:31:11 : Start Power Parse

The class file is blank, race has 6 entries and powers has the usual stuff. If I remove the entries from powers (leaving it blank) it parses okay, but even a single entry causes the crash.

any ideas?

[Griogre]

It's likely some format changed in the WotC DDI database. I'll take a look at it when I get time later.

[tdewitt274]

To whom it may concern...

I'm going through and parsing the while D&D Compendium (hopefully). I'll be marking the ones that I have errors with to come back and address later. How do you want me to report any errors I come across, if you are interested? Would anything help in the process?

I'm planning on making notes in an error file with possible resolutions. I'm also planning on adding a link to the appropriate Compendium entry. If something makes it easier in the troubleshooting process, let me know.

On an unrelated note, is there a template somewhere that I can use to fill in the "blanks"?

Thanks!

Todd

[Griogre]

I'm attempting to parse powers from the Dungeon Survival Handbook.
As far as I know the Dungeon Survival Handbook is not in the DDI database. How are you selecting it?

[Griogre]

I'm having a problem parsing Monster Manual 1;

The last lines are
12/12/2012 12:31:11 : ERROR: line:
12/12/2012 12:31:11 : Start Power Parse

The class file is blank, race has 6 entries and powers has the usual stuff. If I remove the entries from powers (leaving it blank) it parses okay, but even a single entry causes the crash.

any ideas?
I took a look at it and when I loaded in the parse settings for MM1 found I don't bother to parse the monster powers. :p So you are correct, I don't know how long its been broken, I got the same error as you when I did try it. I found I didn't parse the rituals either but they still work when I checked. You are just going to have to skip parsing the monster racial powers unless you want to mark the powers up by hand.

@tdewitt274 Generally, the most common errors with parsing the scrape are html tags in the scrape text file. If after you scrape a book, you search for a < in the text file and when you find a tag, mass find and replace the entire tag including the < and > and replace with a space. Sometimes there are more than one tag in a file so make sure you get all of them, usually this is a problem in the powers.txt, npc.txt and traps.txt files if a book produces those. *Don't* do this in rituals.txt its suppose to be full of XML elements.

[judesudbury]

As far as I know the Dungeon Survival Handbook is not in the DDI database. How are you selecting it?

Hi, from the dropdown - it allows you to select it (although it starts: "Into the Unknown: "). Scrapes fine, but won't parse.

Looks like I'm going to have to input stuff by hand. :cry: Fortunately I managed to find a bunch of the main modules pre compiled online, so will have to ask my players to only choose vanilla powers and the like.

There's a great item parser that's available from another member, but unfortunately it doesn't support powers...

Thanks for looking
Jude

[judesudbury]

Good luck with that tdewitt274, sounds very ambitious though!

[Griogre]

Hi, from the dropdown - it allows you to select it (although it starts: "Into the Unknown: "). Scrapes fine, but won't parse.

Ah, I was looking for Dungeon Survival Handbook. :p I scraped it and took a look at the powers. The parser isn't going to be able to handle them in their current format. I don't have/use that book, but from reading the powers the format seems very different on a number of powers. If that book is important to you you could mark powers up by hand. The parser just reads data files and in its Documentation there are samples on how to do this. The hard part of doing your own stuff is getting the data and it does look like the scraper did that so it would mostly be a matter of just putting in the control character sequences.

I'd scrape the all the PH's and Splats first so you would have a lot of examples and then read the parser documentation before trying to mark up the powers in the DSH. Make sure you keep all your text files and save the settings for each book with them.

[Griogre]

Hi, from the dropdown - it allows you to select it (although it starts: "Into the Unknown: "). Scrapes fine, but won't parse.

Looks like I'm going to have to input stuff by hand. :cry: Fortunately I managed to find a bunch of the main modules pre compiled online, so will have to ask my players to only choose vanilla powers and the like.

I just updated most of the stuff I allow in my game last month so I'm pretty sure you can still scrape the all the PHs, Essential PHs, and Splats. I'd suggest you do that because unless those modules are recent there may have been a lot of changes (especially if those are those old RPGA modules). They did a major balance update to all powers/feats around the first quarter of this year.

[judesudbury]

Ah - ok, thanks. Just taking a look at the documentation - shouldn't be too painful adding in the markup. By the way - what are 'Splats'?

[Griogre]

Splats = Splat Books = Books that increase PC power levels so monsters go "splat!" :) They are always optional books that are not part of the core rules.

In 4E I was referring to the Arcane, Divine, Martial, Primal and Physic Power books as the older splat books, though I suppose Heroes of Shadow, the Feywild and the Elemental Chaos are splat books as well. I parsed all of these books recently.

[tdewitt274]

Good luck with that tdewitt274, sounds very ambitious though!
Meh, doesn't take too long if you know how to cheat ;) About 5 hours, and that was with researching the "cheats". I have some errors to wade through, but it went well.

Here's hoping it's all right!

[judesudbury]

Splats = Splat Books = Books that increase PC power levels so monsters go "splat!" :) They are always optional books that are not part of the core rules.

In 4E I was referring to the Arcane, Divine, Martial, Primal and Physic Power books as the older splat books, though I suppose Heroes of Shadow, the Feywild and the Elemental Chaos are splat books as well. I parsed all of these books recently.

Thanks for the explanation Griogre, I'll get scraping then!

[askaval30]

sorry to exhume an old thread, but I was just trying to parse the DMG2 and for some reason the parse seems to hang up on the 'Beginning Compression' line. Did a new version of the parser come out recently?

(I'm using Eugene's 118 upgrade to 4.1.2.3)

[Griogre]

Using Egenez beta, I just scraped and parsed the DMG2 a few minutes ago. It seemed to work fine after I replaced the html tags with spaces in traps. What was your error message? Did you turn off items?

[askaval30]

did it with both Items on and off and it still hangs up on compression... odd cause I've tried it on 2 separate PCs and it worked a month ago

[Griogre]

Could it be because you are running FG and the module is open or something so the file is in use and can't be replaced? Or have the xml open in an editor? Compression is were it is actually saving the module.

[Zeus]

Just a thought, check the temp path you have set is not full of any additional large files, this maybe causing the ZIP stage to take a long time leading you to believe its hung when in fact its just busy.

The Parser ZIPs the total content of this folder.

You should only have the following files in the temp path when the Parser has completed its parsing and is ready to ZIP.

- definition.xml
- [client|common|db].xml
- images (folder)
- tokens (folder)
- thumbnail.png

[askaval30]

no additional instance of the module was open, however temp path might be an issue, I'll try reparsing from a clean dir and just wait to see if it goes through. As always I am grateful and amazed at the speed of the help we get on these forums. kudos!

[askaval30]

no additional instance of the module was open, however temp path might be an issue, I'll try reparsing from a clean dir and just wait to see if it goes through. As always I am grateful and amazed at the speed of the help we get on these forums. kudos!

UPDATE: half an hour stuck on beginning compression, so I guess it is going nowhere. must be something on my system if no one else has had this problem crop up