Generating Modules with External Tools

[tpkurilla]

Greetings, all.

I've been searching for any documentation on how to generate modules through means other than exporting from FG itself.

Obviously it is possible because that is what Tenian's parser does.

Just as obviously, I just can't seem to find the documentation. Probably has something to do with how tired I usually am when I can finally do a little searching :D

Can someone please point me in the right direction? Any words of wisdom or lessons learned would be greatly appreciated as well.

Thanks in advance,

-Thomas

[Griogre]

Usually, you need to understand how an individual ruleset expects its XML data to be structured and just build the XML files accordingly. That would normally be somewhat ruleset specific though many rulesets do things similarly. Traditionally, I would say most people learned by taking apart an existing library module and looking at the XML. Module files are just plain zip files so you can take them apart by changing the .mod file extension and renaming it .zip and just then just unzipping the files. Some zip programs will recognized the .mod files as zip files as well.

The good news with 4E is that the 4E parser also can a lot of stuff for you so may find it easier to use instead of entering things in an XML editor.

Depending on what you are trying to do it may be easier to use the 4E parser to build your module. What are you trying to do?

[tpkurilla]

Honestly, I was hoping to become proficient enough to eventually provide the follow-on replacement for Tenian's parser.

It has been crashing on items for me for months, and appears that the next release will likely be the last.

Given all that, I figured that I could at least give it a try. I've already got the ability to get all of the data from the DDI web site. Now I just need to see if I can figure out the mapping FG expects for 4E.

Tenian's parser has been a tremendous god-send, but it seems like its days are numbered. Anything that I come up with will be released GPL so we'll have support even if I can't continue at some point.

Guess it is time for me to roll up my sleeves, and start disecting the modules that were created by Tenian's parser.

-Thomas

[Griogre]

Out of curiosity what language are you using to write your scraper/parser?

[tpkurilla]

Out of curiosity what language are you using to write your scraper/parser?

I'm starting with a GPL program written in C++. The program is actually pretty well written and documented, so I will not likely change languages.

Although, since I am wanting to introduce an intermediate step of actually saving the scraped text, the parser could be in something else.

[Griogre]

If you are just planning to do 4E then you could still use Tenian's parser to build 4E modules. The reason I asked on the language is I recently wrote a C# reparser for the monster modules because I prefer a simpler and flat organization of monster library links.

If you are planning on supporting 5E then you would be better off writing you own parser probably.

[tpkurilla]

As stated above, I've been very appreciative of Tenian's parser. In the last few months, however, the support has not kept up with the changes. I have been unable to scrape items reliably since MME was added to DDI.

My goal is not to keep using a parser that is very clearly losing all support in the very near future. There is no guarantee that the last update we're hoping for in the coming week will address the issues I've been having.

My goal is to write a parser that will be open and available to the community, so that 4e, and eventually 5e, support is maintained.

I'm hoping that I'll be able to provide something useful without too much stumbling. Tenian's parser has been an amazing utility. If you've looked at his XML output you can't fail to appreciate his work. It truly is very well structured, and very easy to read and understand.

I would rather have been able to tweak what he's got as a base. But that seems exceedingly unlikely.

So I'm doing what I can. Hopefully I'll actually be able to provide something that the community will find useful enough to use.

-Thomas

[Torgaard]

My goal is to write a parser that will be open and available to the community, so that 4e, and eventually 5e, support is maintained.

Would it be weird if I started squeeling gleefully? Kind of an "eeeeeee!"?

Dude, that would be awesome if you can get make that (a new parser) happen. Sounds like a big job, but MAN is (was) that parser useful. Frankly, it kinda took the wind outta my sails when I read that Tenian's is not long for this world.

Good luck! Hopefully it all works out.

[Griogre]

tpkurilla, I wasn't entirely sure where you wanted to go in the future and was just wanted to point out its the DDI scraper that's the broken part of the parser. The part that converts the text files to XML works fine and is likely to continue working for a very long time since the 4E ruleset's XML data structure is not likely to change as far as library module data goes.

Depending on how you want to do your development and how much time you have to work on it, you could do a scraper first that outputs text files such that Tenian's parser could make the XML and 4E modules. Then you would write your own parser aimed at 4/5E.

The scrapper is always going to be fragile and be where the maintenance work is since WotC controls the data base and doesn't have any interest in helping FG users or keeping the data base consistent right now. They may even view FG as a rival if they ever get Gametable out of beta. This might be a good reason to keep the parser and scraper parts separate.

[Moon Wizard]

There are a couple minor updates to the data structure with the v2.9 release, but nothing major.

JPG