Pathfinder Creature Parser - Alpha release

New v2 beta test available here: https://www.fantasygrounds.com/forum...2-Beta-Version

UPDATE - 3rd January 2012. Full Version 1.0.0.15

Attachment 2247

New features/items fixed:

• Modified to be able to read stat blocks from the Combat Manager application - https://www.combatmanager.com/. See readme.txt file and below.
• logging disabled by default. Parsing is much faster without logging!
• added custom xp tag to each create - XP is not used by Fantasy Grounds at present.
• Added a token reference for each creature - see readme.txt and below.

Note: This is just a creature parser, it will not parse traps or vehicles. Do not include stats for traps or vehicles in the statblock file you parse as they will break the parser!

Combat Manager
https://www.combatmanager.com/

Check this thread for more details.

NOTE: Minor text formatting is still needed with statblocks copied from the monster statblock within combat Manager:
1) Move description to the beginning of the statblock.
2) Next line after description must be <NAME> CR <CR Value> all on one line. For example: Ghoul CR 2.
NOTE: When doing a "Select All" and copy from Combat Manager, the CR entry will be placed on the line before the name - move this to the end of the name line.
3) Remove all blank lines in the statblock.
4) Leave a blank line between each creature statblock.

Token Reference
A token reference has been added for each creature - referencing the creature display name in the tokens directory in the module - with a .png extension. For example:

Creature: Archon, Shield
Token: tokens/Archon, Shield.png@Bestiary2

Note: this needs the manual addition of the image to the "tokens" directory in the module and any required tokens - named <creature name>.png


GOTCHAS

Pathfinder Society - creatures in PFS modules don't have XP values. The Parser relies completely on the XP line between creature Name/CR and Type to be able to split these lines up.

Workaround: PFS creatures, add a new line: "XP 1" (without the quotes) between the Name/CR line and the type line/s.

Stat block titles
The Parser mainly relies on the stat block main titles to correctly read the data correctly. Make sure that when you copy/paste the creature that the main titles are one a line by themselves, no space in front and no spaces in the title (I've seen spaces appear when copying some creatures). The titles are:

Defense
Offense
Tactics (if present)
Statistics
Ecology (if present)
Special Abilities (if present)

Frog God Games Statblocks: Frog God Games products (Slumbering Tsar, Rappan Athuk, Razor Coast, etc.) usually adhere to the Paizo statblock format - with one exception: the "Stat block titles" are not present, the books contain separator lines instead - so you must enter the relevant "Stat block titles" as listed above in place of the separator lines.

HeroLab statblock output
In HeroLab go to File -> Output Active Hero Statblock -> Plain Text -> Save to File and the output parses fine with some minor changes to the HeroLab exported file:

1. Add an XP line between the Name/CR line and Type line.
2. Remove the --------- lines around the section titles. (remove the whole line)
3. Remove any blank lines within each individual statblock.
4. Remove the HeroLab notice at the bottom.

-------------------------------------------------------------


Hi everyone,

For a while I've been slowly developing an application that allows parsing of Pathfinder creature stat blocks from a text file into a Fantasy Grounds mod file.

I've tried to design this so that stat blocks can just be copied from PF material (start with the description and end with the special abilities) into a text file and then the text file can be parsed into a creature module for loading in FG.

See the files in the attachment. Extract to a folder. Run setup.exe.

Follow the instructions in readme.txt.

Please let me know if it works and if you have any issues... A good place to look to see how the parsing is going is a temporary file created during parsing - it is the same name as your text file with an extension of .fpf; or check the logfile (same name as your text file + .log). The .fpf file contains formatted stat block info - it might help you to spot formatting errors in the original text file (I've seen a few from copy/paste from the Bestiary 2) and allow you to fix them yourselves and get some creatures parsed!

If you do have issues, please PM me with details of:

1) The error raised.
2) The text stat block for the creature you're parsing. (The one that caused the error may be indicated in the text line output).
3) The .fpf file entry for this creature and the logfile.

Any specific formating issues we should look for when creating the text file (doesnt like certain chracters ior spacing iussues?). I ran an npc from the first steps scenario into the parser and received the following parser notice toward the end:

Processing creature: Deandre Dulay
Error in ReadWriteFormattedCreature, Message: Index was outside the bounds of the array.
Error in btnStartParse_Click, Message: Conversion from string "Index was outside the bounds of " to type 'Integer' is not valid.
Module file created: C:\Documents and Settings\Main\My Documents\Downloads\pathfinderSocietyScenarioIntro 1FirstStepsPartIInServiceToLorePFRPGPDF\parsermodt est.mod


I'm assuming keeping the tactics in the text file was a bad idea, but anything else that would specifically lead to the 2 errors above? I get a mod file and it shows in my library, but Im unable to see anything inside when I open. One other basic question for now - should I be encoding the .txt file as an ANSI, Unicode or UTF-8 file?

Quote:

---

Originally Posted by Talen

Any specific formating issues we should look for when creating the text file (doesnt like certain chracters ior spacing iussues?).

---

I've coded remove spaces at the end and beginning of lines, remove TABs, remove long hyphens and replace with the minus (-) sign, but I'm sure there are some in there that I still need to track down - I noticed the "x" for "x3" damage wasn't processing properly and need to update that.

But, I tried to design it so that you can just copy and paste the stat block - have a look at the example in the zip file.

Quote:

---

Originally Posted by Talen

I ran an npc from the first steps scenario into the parser and received the following parser notice toward the end:

Processing creature: Deandre Dulay
Error in ReadWriteFormattedCreature, Message: Index was outside the bounds of the array.
Error in btnStartParse_Click, Message: Conversion from string "Index was outside the bounds of " to type 'Integer' is not valid.
Module file created: C:\Documents and Settings\Main\My Documents\Downloads\pathfinderSocietyScenarioIntro 1FirstStepsPartIInServiceToLorePFRPGPDF\parsermodt est.mod

I'm assuming keeping the tactics in the text file was a bad idea, but anything else that would specifically lead to the 2 errors above?

---

One of the first things I need to do is try to make the error messages more meaningful!

I did all of my testing against monsters from Bestiary 1 and 2 and followed the stat block definition from page 6 of Bestiary 2. Late last night I ran a couple of NPCs from Carrion Crown 2 (Trial of the Beast) through and realisesd that I'm going to have to do some more work as NPCs are different to Monsters! :mad:

As a starter Tactics need to be removed for now. I'll code for that later and move the tactics into the "Other" tab of the NPC.

Please send the .txt file you were using to martinblake1415 (at) h o t m a i l (dot) c o m and I'll take a look. Sorry it flopped at the first step!

Quote:

---

Originally Posted by Talen

I get a mod file and it shows in my library, but Im unable to see anything inside when I open. One other basic question for now - should I be encoding the .txt file as an ANSI, Unicode or UTF-8 file?

---

Even with errors I continue processing (in case you're doing a lot of creatures together and some of them work). If there was an error on the first creature then there won't be a complete db.xml file in the mod.

The files I've been using are ANSI with Windows 1252 encoding. The encoiding has been necessary for some of the above mentioned issues (hyphens etc). I may look into making it cover other formats at a later date (if it's relatively easy and not expanding the formats is an issue to some people).

Thanks for the info - I sent the text file your way.

Update: V 1.0.0.7 fixes items 2 and 3.

Issues identified so far:

Thanks to Talen for pointing me in their direction.

1. PFS NPCs don't have XP entry. A lot of the initial logic in determining description, name/CR and type needs this line. Workaround - enter XP 1 in a line between name/CR and type.
2. Creatures with "gear" entries rather than treasure. Workaround - change the Gear/Combat Gear title to "Treasure".
3. Tactics section causes errors. Workaround - remove all tactics section (upto but not including "Statistics" title).

I'm working on finding good solutions to the above items. In the mean-time if you want to parse the creature/NPC use the workarounds mentioned.

Update available - see the first post for download, changes and gotchas.

Working great! The time save is fantastic! A few nitpicking points at the moment. The parser still doesnt like seeing "other gear"...it comes out as "no match" in the npc - but the good news is that it doesnt stop the parse and doesnt make a practical difference. Also, Im sure this was a further down the road point - but traps will stop the parse if you try to get them in.

Quote:

---

Originally Posted by Talen

Working great! The time save is fantastic!

---

Glad it's working for you. :) Yeah, I've been very surprised with the time save - I imported all the custom monsters from Carrion Crown 2 in about 5 minutes! :D

I'm currently around 30% into a first pass of Bestiary 2 - currently up to Dragons (there are a lot of A's and D's in Bestiary 2!). So far, everything parses through OK - a couple of things to sort out, the main one being single attack format...

Quote:

---

Originally Posted by Talen

A few nitpicking points at the moment. The parser still doesnt like seeing "other gear"...it comes out as "no match" in the npc - but the good news is that it doesnt stop the parse and doesnt make a practical difference.

---

Looks like "Gear" slipped through my "do comparisons case insensitive" code. I'll update in the next release. Before then, if you want to avoid "no match" appearing, use "Gear", "Combat Gear" and "Other Gear" (leading capitals on all).

Quote:

---

Originally Posted by Talen

Also, Im sure this was a further down the road point - but traps will stop the parse if you try to get them in.

---

Yep, I'm sure anything other than creature stat blocks will break the parser. I've no plans to bring in traps - basically, because there is nowhere to put them in Fantasy Grounds! I suppose they could be put in as freeform story entries, but you may as well copy/paste them directly into FG yourself...

I've done some testing with the HeroLab statblock output (File -> Output Active Hero Statblock -> Plain Text -> Save to File) and this parses fine with a couple of minor changes to the HeroLab exported file:

1. Add an XP line between the Name/CR line and Type line.
2. Remove the --------- lines around the section titles. (remove the whole line)
3. Remove the HeroLab notice at the bottom.

Is the hero lab use just a validation or is there some other use?

Quote:

---

Originally Posted by Talen

Is the hero lab use just a validation or is there some other use?

---

Nope, its not just validation. On the HeroLab supports FG threads a few people were asking for HL and the FG character importer to support NPCs/creatures.

Now you can create custom creatures and bad add NPCs in HL and import them to FG with this parser. :)

I'm not sure if this is just an error on my part so I thought I would ask. I have tested both a Hero Lab stat block and a sample monster from Bestiary 2 with no issues parsing either using the instructions provided.

The issue I am having is both creatures AC are correct in the module however both have their AC changed to 10 when I place them on the combat tracker.

Quote:

---

Originally Posted by Koryn

I'm not sure if this is just an error on my part so I thought I would ask. I have tested both a Hero Lab stat block and a sample monster from Bestiary 2 with no issues parsing either using the instructions provided.

The issue I am having is both creatures AC are correct in the module however both have their AC changed to 10 when I place them on the combat tracker.

---

Hi Koryn - no, it's not an error on your part; it's on mine! The AC line in the library entry has "AC" at the beginning - that's causing FG a problem. I'll fix that in the next release of the parser (which will be out soon).

When errors do arise, how do you pinpoint the spot with the error occurred using the log file? Does the log pinpoint the creature where the first error occurred?

Quote:

---

Originally Posted by Talen

When errors do arise, how do you pinpoint the spot with the error occurred using the log file? Does the log pinpoint the creature where the first error occurred?

---

Search for an entry "FormatDataFile: Reading and formatting creature stat block:" it will tell you the name/CR line from the text file. Then all the lines after that should relate to that creature.

You should see something like this in the log at the beginning of each creature stat block:

Code:

---

7/10/2011 10:33:03 PM: Entered Sub FormatDataFile.
7/10/2011 10:33:03 PM: FormatDataFile: Read past starting blank lines (if any).
7/10/2011 10:33:03 PM: FormatDataFile: Opened StreamWriter.
7/10/2011 10:33:03 PM: FormatDataFile: Written DESCRIPTION line: A shimmering, colorless mass congeals to form a four-armed humanoid shape with an eye-like pattern in its torso.
7/10/2011 10:33:03 PM: FormatDataFile: Written NAME line: Aeon, Bythos CR 16
7/10/2011 10:33:03 PM: FormatDataFile: Reading and formatting creature stat block: Aeon, Bythos CR 16
7/10/2011 10:33:03 PM: FormatDataFile: Written XP line: XP 76,800
7/10/2011 10:33:03 PM: FormatDataFile: Written TYPE line: N Large outsider (aeon, extraplanar)

---

Thanks for the feedback Talen and Koryn. There's a new release available (see first post) that addresses the AC issue and has some other minor improvements.

Please keep feedback coming - I'd like any issues reported so that we can get this parser working for 99.9% of creature stats... :)

Just wanted to give you guys a big thanks for a. Coding this and B. Testing it out, I know it will be a real boon for running adventure paths, and having the Bestiary 2 available to use as well.

Everything I have thrown at it is working great after the last update. I have been doing some testing with Tome of Horrors with success after a bit of editing to the text doc pre parse.

My thanks to you Trenloe for the work you have put into this. It is greatly appreciated.

Quote:

---

Originally Posted by Koryn

Everything I have thrown at it is working great after the last update. I have been doing some testing with Tome of Horrors with success after a bit of editing to the text doc pre parse.

---

Great to know it's working well for you.

Tome of Horrors was going to be my next check - what editing did you have to do to get it to work?

EDIT: Actually, I see what is up. The Tome of Horrors doesn't use the section headings that normal stat blocks do (Defense, Offense, Statistics, Ecology and Special Abilities) - the parser relies heavily on these headings to be able to extract information across multiple lines with some sub-headings sometimes being there and other times not (e.g. Aura appears rarely).

So, for other people's information, Tome of Horrors creatures will need the section headings putting into the stat block - Koryn, I'm sure that's what you've already found out...

Quote:

---

So, for other people's information, Tome of Horrors creatures will need the section headings putting into the stat block - Koryn, I'm sure that's what you've already found out...

---

The only thing I had to do other than add in the headings was change the semicolons on the saves line to commas. Perfect parse after those two things are done.

I was perusing the legal appendix at the back of ToH and pondering whether materials in the book could be compiled for general use. Not sure if this is something each user would need to do for themselves.

For Bestiary 2 - apart from the Offense spacing issue - what other changes di you have to make if any with the new parser?

Quote:

---

For Bestiary 2 - apart from the Offense spacing issue - what other changes did you have to make if any with the new parser?

---

I haven't had to make any changes to the Bestiary 2 creatures using the newest version of the parser (1.0.0.9).

Quote:

---

Originally Posted by Talen

For Bestiary 2 - apart from the Offense spacing issue - what other changes di you have to make if any with the new parser?

---

Sometimes copying out of the Paizo PDF spaces also appear in the CR (i.e. "C R") which will break the parser - as checking for " cr " is a major part of the initial parsing - as it's the first consistent text that is present in the creature stat block.

Out of curiousity: Where are you copying the text from a pdf or the paizo web site ?

I think the Bestiary 2 is on the web site correct ?

https://paizo.com/pathfinderRPG/prd/...sterIndex.html

After you do the bestiary 2 will you make it available ?

I'm copying from the PDF. You can use the PRD on the website, but you'll have to do additional formatting - remove all blank lines and the lines around the section titles.

I'm parsing creatures from Jade reagent and getting some issue's with 2 of them I can't figure out... here is the text... scratch that text sent in pm..

Quote:

---

Originally Posted by Talysian

I'm parsing creatures from Jade reagent and getting some issue's with 2 of them I can't figure out... here is the text... scratch that text sent in pm..

---

Hi Talysian - thanks for trying the Parser and for forwarding on details of your issue.

For reference of others: there are typo errors in some of the Jade Regent creatures. They are missing a comma (,) after the strength statistic. For example:

STATISTICS
Str 1 Dex 18,

Which should actually be:

STATISTICS
Str 1, Dex 18,

I haven't looked at the Jade Regent creatures too much as I'm probably going to be a player in that adventure path soon. Just check that each creature has properly formatted statistics and you should be OK.

Quote:

---

Originally Posted by wbcreighton

After you do the bestiary 2 will you make it available ?

---

Yes, I'll make it available as a module. I've been away at PiazoConOz playing lots of Pathfinder so haven't been able to improve on my 30% completion. Now I'm back I'll continue on the copy/paste-fest that is needed... :)

So in playing "Where's Waldo", where's the error in the following text file?

Giant Amoeba CR 1
xp 100
N Small ooze (aquatic)
Init –5; Senses blindsight 30 ft.; Perception –5
Defense
AC 6, touch 6, flat-footed 6 (–5 Dex, +1 size)
hp 15
Fort +3, Ref –5, Will –5
Defensive Abilities ooze traits
Offense
Speed 10 ft., climb 10 ft., swim 20 ft.
Melee slam +3 (1d3+1 plus 1d3 acid and grab)
Space 5 ft.; Reach 5 ft.
Special Attacks constrict (1d3+1 plus 1d3 acid)
Statistics
Str 12, Dex 1, Con 16, Int —, Wis 1, Cha 1
Base Atk +1; CMB +1 (+5 grapple); CMD 6 (can’t be tripped)
Skills Climb +9, Swim +9
SQ amphibious

Quote:

---

Originally Posted by Talen

So in playing "Where's Waldo", where's the error in the following text file?

Giant Amoeba CR 1
xp 100
N Small ooze (aquatic)
Init –5; Senses blindsight 30 ft.; Perception –5
Defense
AC 6, touch 6, flat-footed 6 (–5 Dex, +1 size)
hp 15
Fort +3, Ref –5, Will –5

---

There's no hitdice on the HP line, make the line: hp 15 (2d8+6)

If you didn't know, the giant Amoeba is in the Bestiary 2 module too! :)

Quote:

---

Originally Posted by Trenloe

Yes, I'll make it available as a module. I've been away at PiazoConOz playing lots of Pathfinder so haven't been able to improve on my 30% completion. Now I'm back I'll continue on the copy/paste-fest that is needed... :)

---

Bestiary 2 mod available for download here.

:)

Quote:

---

Originally Posted by Trenloe

There's no hitdice on the HP line, make the line: hp 15 (2d8+6)

If you didn't know, the giant Amoeba is in the Bestiary 2 module too! :)

---

I did know that (and thank you)- but trying to pull myself up by the bootstraps and figure out where my mistakes are. I still cant efficiently find them with the log, but, I did learn the following:

1. As you identified, hp need the hit dice entry as well.
2. Defense saves need to be on the same line.
3. You cannot have 2 "Treasure" entries using the same label.
4. Related to the above - you cant have a colon after the "Treasure" label.
5. Senses need to be separated by a semicolon.

In the Bestiary2, I've come cross the slam attack entry for the Giant Amoeba which did not parse the attack. The funny thing is that when I parsed the same creature entry from a different source (the First Steps Part 2 scenario) - it also did not parse the attack properly. Is the "and grab" description gumming it up?

Quote:

---

Originally Posted by Talen

In the Bestiary2, I've come cross the slam attack entry for the Giant Amoeba which did not parse the attack. The funny thing is that when I parsed the same creature entry from a different source (the First Steps Part 2 scenario) - it also did not parse the attack properly. Is the "and grab" description gumming it up?

---

Hey Talen - when you say didn't parse the attack are you saying that there was an error caused or that the attack string isn't being recognised by Fantasy Grounds?

The current B2 mod has parsed has the attack listed as: "slam +3 (1d3+1 plus 1d3 acid and grab)". If you change the "and grab" to "plus grab" the attack should work.

I'll keep an eye out for attacks like this and see what I can build into the parser.

I mean fg won't make the attack ( it does parse in the sense that it shows up in the entry).
I'll keep an eye out for those too.

Quote:

---

Originally Posted by Talen

I mean fg won't make the attack ( it does parse in the sense that it shows up in the entry).
I'll keep an eye out for those too.

---

Yeah, it's an FG thing. The attack doesn't work with 2 damage types listed as "x and y" e.g. "1d6 fire and acid" I had to manually change it to "1d6 fire, acid". So, unfortunately, it's not just a case of changing any "and" in the attack line to plus...

I'll compile a few examples and see what the options are and see what is best for importing into FG...

My trouble shooting skills are on the fritz, so looking for an assist - anyone know where Im missing the error here? I thought it might be the sneak attack damage or the racial modifiers, but changing the format of the attack and removing the racial modifiers didnt yield any results.

Spriggan High CR 3
xp 100
CE Small humanoid (gnome)
Init +4; Senses low-light vision; Perception +7
DEFENSE
AC 17, touch 15, flat-footed 13 (+2 armor, +4 Dex, +1 size)
hp 22 (4d8+4)
Fort +5, Ref +5, Will +1
OFFENSE
Speed 20 ft.
Melee mwk morningstar +5 (1d6–1)
Ranged light crossbow +8 (1d6/19–20)
Special Attacks sneak attack +2d6
Spell-Like Abilities (CL 4th; concentration +4)
At will—flare (DC 10), scare (DC 12), shatter (DC 12)
TACTICS
During Combat The spriggan grows to Large size as soon as
possible and lumbers into combat, concentrating on the
most dangerous opponents (see below for Large statistics).
The spriggan does not much like the atomies, and so avoids
harming PCs that seem to be concentrating their attacks
against the diminutive duelists.
Morale The grim spriggan expects and offers no quarter. He
fights until slain.
STATISTICS
Str 9, Dex 19, Con 12, Int 10, Wis 10, Cha 9
Base Atk +3; CMB +1; CMD 15
Feats Combat Reflexes, Weapon Focus (morningstar)
Skills Climb +1, Disable Device +11, Perception +7, Sleight of
Hand +11, Stealth +15; Racial Modifiers +2 Climb, +2 Disable
Device, +2 Perception, +2 Sleight of Hand, +2 Stealth
Languages Aklo, Gnome
SQ size alteration, spriggan magic, spriggan skills
Gear leather armor, light crossbow with 10 bolts, masterwork
morningstar
SPECIAL ABILITIES
Size Alteration (Su) At will as a standard action, a spriggan
can change his size between Small and Large. Weapons,
armor, and other objects on the spriggan’s person grow
proportionally when he changes size (objects revert to
normal size 1 round after a spriggan releases them). When a
spriggan becomes Large, his speed increases to 30 feet, he
gains +12 Strength, –2 Dexterity, and +6 Constitution, and
he takes a –2 size penalty to his AC. While Large, a spriggan
cannot use his sneak attack or his racial spell-like abilities.
Spriggan Magic (Ex) A spriggan gains a +1 racial bonus on
concentration checks and to save DCs for all of its racial
spell-like abilities.
Spriggan Skills (Ex) Climb, Disable Device, Perception, Sleight
of Hand, and Stealth are class skills for spriggans.

Quote:

---

Originally Posted by Talen

My trouble shooting skills are on the fritz, so looking for an assist - anyone know where Im missing the error here? I thought it might be the sneak attack damage or the racial modifiers, but changing the format of the attack and removing the racial modifiers didnt yield any results.

---

This one's a hard one - it's actually the re-use of the word "skills" in the special abilities section. Parsing out the data from the Statistics section of the statblock is a difficult task as there can be a lot of freeform text here and a lot of the usual headings can be missing so it's harder to know for sure where one section starts and another begins. Although, I think I have an idea how I can do this a bit better - I'll do this is a future update.

For now, you can get this create to parse by changing the Special ability "Spriggan Skills (Ex)" to Spriggan Skill (Ex)" - so the parser won't think there are multiple "skills" headings in the stat block.

New version now available in post #1.

I've been doing a lot of work behind the scenes ironing out a few wrinkles. I now feel that this is ready to be called a full release (not an Alpha release) - I've jumped straight past any Beta releases! :D

See post #1 for new features - and the changelog in the readme.txt in the installation zip file.

Also - get the updated Bestiary 2 module here. It's been created with the latest parser (and a bug fixed statblock file) so this should be much nicer and more accurate to use.

UPDATE - 16th November 2011. Full Version 1.0.0.12

New features/items fixed:

• Multiple attacks formatted into single attacks - full implementation.
• Cancel parse button.
• Massively increased logging.
• Disable logging option (handy on slow PCs or when parsing large creature lists).
• Will process if no HD entry pressent on HP line.
• Processing of saves with ; seperator (Tome of Horrors). Standard seperator is a comma (,).
• Process headings with a colon at the end of the label.
• Handle saves with additional multipliers - e.g. +4 vs. fear.
• Handle parsing of "Spell-like" abilities - Tome of Horrors.
• Process weaknesses line and place in Special Qualities (SQ).
• Rename "vulnerable" weaknesses to "vulnerability" - Fantasy Grounds will not recognise "vulnerable".
• Much better formatting of special abilities in the "other" tab.

I was just wondering no that you have worked through doing the Monster Parser, Is there any chance for a spell parser (PLEASE) Feat Parser (Please)

Would it be Worth it to do.

Just curious, Thank You For everything you have done so far.

P.S. and thanks for the cool stuff to come(as I know it will)

Madman

Chris