Status of Large Migrated from FGC folder

[dellanx]

I realize that this Devs are aware of this issue. I will try my best to describe my observations, even though I can’t provide code or any such thing:
1. Loaded FGU took about 2 minutes, felt like an improvement, got the following log, I am not loading any extensions except Advanced Bistiery.:

a. [12/19/2019 9:12:54 AM] [NOTICE] Launcher scene starting.
b. [12/19/2019 9:13:05 AM] [NOTICE] Daily session backup created.
c. [12/19/2019 9:13:06 AM] [<color="red">ERROR</color>] Database restore point found. Saved as new backup file (C:/Program Files/SmiteWorks/Fantasy Grounds/FantasyGrounds_Data/campaigns/End of the World/db.backup.637123435850677611.xml)
d. [12/19/2019 9:13:06 AM] [NOTICE] Spawning private server.
e. [12/19/2019 9:13:06 AM] [NOTICE] Connected to game server.
f. [12/19/2019 9:13:06 AM] [NOTICE] Launcher scene exiting.
g. [12/19/2019 9:13:06 AM] [NOTICE] Tabletop scene starting.
h. [12/19/2019 9:15:37 AM] [<color="red">ERROR</color>] Process Data File (PFRPG GameMastery Guide NPCs): An error occurred while parsing EntityName. Line 1469, position 599.
i. [12/19/2019 9:16:10 AM] [<color="red">ERROR</color>] Process Data File (d20pfsrd Equipment): An error occurred while parsing EntityName. Line 147, position 25.
j. [12/19/2019 9:16:10 AM] [<color="red">ERROR</color>] Process Data File (d20pfsrd Feats): An error occurred while parsing EntityName. Line 147, position 25.
k. [12/19/2019 9:16:10 AM] [<color="red">ERROR</color>] Process Data File (d20pfsrd Traits): An error occurred while parsing EntityName. Line 147, position 25.
I updated before I loaded FGU. Went to click on box and FGU crashed. Before FGU crashed after I tried to open one of the interfaces like “NPC” or “STORY”. Now it just crashed, so in that respect things are worse.
I am using Pathfinder, in Windows 10.
List of Modules in the Modules folder: attached below
30912

Notes:
1.0 Anlira.mod is 172 MB player made module attempted migrated – would not load in previous attempts,
Transfer.mod is 18.9 MB player made module migrated – has loaded before, not sure now,
Varisia.mod 972 MB player made module attempted migrated – would not load in previous attempts.
Thanks so much

[Trenloe]

There are errors reported in other modules - looks like community created modules that probably don't adhere to the more strict XML that FGU expects. Those modules will need updating to be compatible with FGU.

Delete modulestate.xml from the campaign directory and then load your campaign - this will load up without opening any modules.

[dellanx]

There are errors reported in other modules - looks like community created modules that probably don't adhere to the more strict XML that FGU expects. Those modules will need updating to be compatible with FGU.

Delete modulestate.xml from the campaign directory and then load your campaign - this will load up without opening any modules.

Trenloe, thanks FGU loaded. As I saw before Transfer.mod is visible in the Library. 1.0 Anlira.mod and Varisia.mod can't be seen.

I also know this was asked before, will Gif tokens be allowed in future?

Thanks

p.s. after a few minutes FGU crashed

[Trenloe]

I also know this was asked before, will Gif tokens be allowed in future?
I believe they won't be.

[LordEntrails]

Sounds like varisia.mod is not compliant. Make sure you copied it over correctly, I would do it again just in case.

Animated gifs have been mentioned, no plans but if, then it would be sometime after production release.

[dellanx]

Sounds like varisia.mod is not compliant. Make sure you copied it over correctly, I would do it again just in case.

Animated gifs have been mentioned, no plans but if, then it would be sometime after production release.

Recopied both campaigns, updated same thing as before. Thanks

[dellanx]

Both campaign folders together are over 44 GB. I hope that after product release, if migration of such large campaigns is not possible that at least FGC will still be supported? Thanks

p.s. I do not wish to re-enter everything again, and it seems that at least so far FGU does not support such a large campaign, for reasons beyond my limited abilities to describe.

Also I wonder if there would be a way to send Smiteworks the campaign, perhaps via flash drive or something.

[Neovirtus]

Dellanx, I don't believe anyone was saying that your campaign was struggling to migrate due to its size. If anything, FGU should be better able to handle large file sizes than FGC. From what I read it looked like you had player made modules which are not compatible with FGU without some updating. I would try to load it without any extensions/modules first, get that working, then start looking at what is going on with individual modules separately from the campaign.

[Trenloe]

What @lavoiejh is correct.

But, let's just double check some of the information:


Both campaign folders together are over 44 GB.
Are you saying that two individual campaign directories are 44 GB in size? Or, as you saying that the whole <FG app data>\campaigns directory (containing many campaigns) is that size?


Varisia.mod 972 MB player made module
This is very large for an FG module, be it FGC or FGU. Just what is in that module to make it 972 MB in size? Some large, unoptimized maps? Something else?

[Trenloe]

And also, when you're loading the campaign in FG Unity - have Windows task manager running and monitor the memory usage of the FantasyGrounds.exe process - how much memory does that use, what is the maximum it reaches?

[dellanx]

And also, when you're loading the campaign in FG Unity - have Windows task manager running and monitor the memory usage of the FantasyGrounds.exe process - how much memory does that use, what is the maximum it reaches?

I think basically large non optimized JPG files. I reduced the file size at or below 1 MB each, to use as maps. I have deleted Transfer.mod, 1.0 Anlira.mod and Varisia.mod, FGU now loads and functions. Although as before much data is missing.

With this setup here is a screenshot of Task Manager: 30916

Thanks

[Trenloe]

Although as before much data is missing.
The data from the campaign won't be missing. But data from modules you are no longer loading won't be there.

While monitoring the memory use of FantasyGrounds.exe - open up the modules that contain your data one-by-one - allowing each to fully open and memory use to climb and stabilise before opening another module. This might help to identify which module is causing the issue - and then we can start to look at what that issue may be.

[dellanx]

A short while later FGU crashed, also Google Chrome crashed with it. I could not take a screenshot because Chrome went black, and I had to reboot. Are there any recommendations of how to optimize JPG files? In mass if possible.

[Trenloe]

A short while later FGU crashed
How much memory was it using?

[dellanx]

How much memory was it using? I am not sure, got a "Exclamation Mark" FGU crash image and FGU and Google Chrome crashed. Had to reboot. Before the crash it was climbing, and was at 67%, then crash and screen went black.

[dellanx]

Spent the day optimizing images. FGC seems to be working better, but I dont think it is of any help with FGU. Thanks

[dellanx]

I think the bulk of the campaign is comprised of db.session.xml (https://www.fantasygrounds.com/forums/showthread.php?52833-db-session-files-very-big&p=468433#post468433) files.

[damned]

Back them up and remove all bar the 3 most recent ones

[dellanx]

Dellanx, I don't believe anyone was saying that your campaign was struggling to migrate due to its size. If anything, FGU should be better able to handle large file sizes than FGC. From what I read it looked like you had player made modules which are not compatible with FGU without some updating. I would try to load it without any extensions/modules first, get that working, then start looking at what is going on with individual modules separately from the campaign.

I was having problems with slow performance in FGC as well. So I mad this post (https://www.fantasygrounds.com/forums/showthread.php?52833-db-session-files-very-big) in the House of Healing. I exported the module, then deleted all data but Images, and re-imported the module. Things work better in FGC now, I just need to re-link maps and map shortcuts. However the Modules are not FGU compliant. I don't think any FGC module exported by a player is. 1.0 Anlira.mod and Varisia.mod are not seen by FGU. I have no way of modifying these modules on my own. The newly exported 1 - Part 3.mod is seen, but nothing may be opened.

FYI - in the old days there was no option in FGC to identify the module as a Player Made module like there is now. I believe I could send the modules via Google drive to tech support if necessary.

[Zacchaeus]

What do you mean by a player made module? Modules are made by individuals; they can be players or DMs or neither. Creating a module is a case of entering data in FG and exporting it; or it can be made by just using xml.

Furthermore what do these modules contain?

EDIT: Do you mean a module that a player can access? That is one which defaults to a green tick on the module activation screen? Such modules have always been available to share; but the DM in the past had to specifically do so. Now they default to shared if properly constructed.

[dellanx]

What do you mean by a player made module? Modules are made by individuals; they can be players or DMs or neither. Creating a module is a case of entering data in FG and exporting it; or it can be made by just using xml.

Furthermore what do these modules contain?

EDIT: Do you mean a module that a player can access? That is one which defaults to a green tick on the module activation screen? Such modules have always been available to share; but the DM in the past had to specifically do so. Now they default to shared if properly constructed.


Modules that I made in FGC. I am attaching an 30968. The reason current modules is detected by FGU is that "Player Module" button is checked. 1.0 Anlira.mod and Varisia.mod are not detected. Either way, nothing imports.

[Zacchaeus]

Modules that I made in FGC. I am attaching an 30968. The reason current modules is detected by FGU is that "Player Module" button is checked. 1.0 Anlira.mod and Varisia.mod are not detected. Either way, nothing imports.

Whether a module is a player module (i.e. shared by default) or a DM module (i.e. not shared by default) is very unlikely to have any impact at all on whether the module shows up in FGU (or anywhere else for that matter). Also maodules exported from FG should work since logically they will be in the correct FG format. So I am not sure what exactly is going on and why you are not seeing them. I'm assuming here that when you say they are not detected that you are loading FGU with the correct ruleset and that the modules don't appear in the module activation window?

[dellanx]

Whether a module is a player module (i.e. shared by default) or a DM module (i.e. not shared by default) is very unlikely to have any impact at all on whether the module shows up in FGU (or anywhere else for that matter). Also maodules exported from FG should work since logically they will be in the correct FG format. So I am not sure what exactly is going on and why you are not seeing them. I'm assuming here that when you say they are not detected that you are loading FGU with the correct ruleset and that the modules don't appear in the module activation window?

Yes ~ when check what modules available mine are not there, don't see them.

[Zacchaeus]

Can you put the module somewhere and link it and I’ll have a look at it.

[LordEntrails]

Have you copied the module from FGC to FGU in the operating system?

[dellanx]

Have you copied the module from FGC to FGU in the operating system?

LordEntrails what does that mean? I copied the campaign from the FGC folder and put it in campaign in the FGU folder. I moved modules from FGC module folder into FGU module folder.

[dellanx]

Can you put the module somewhere and link it and I’ll have a look at it.

Sent!

[Zacchaeus]

Alright. So as far as the module 1 - Part 3 is concerned this seems to be a huge bestiary. It is full of hundreds of tokens; most of which are .gif files which is I think the main issue. (I'm surprised they appear in Classic but they do somehow). I tried to open the truly massive client.xml file to see if there was anything wrong with the .xml file but that crashed notepad whenever I tried to do any kind of operation; apart from checking the integrity of the xml which was fine. The NPCs open ok in Classic; it would seem that there are 23 pages of them so that is around 2300 NPC's.

In Unity if you go to assets you will see that the module is visible in assets>tokens. However when you open it you'll get a bunch of errors and most of the tokens are either blank or just don't load. None of the .gif files load so I tried deleting all of the .gif files and rezipped up the module and in Unity the remaining tokens all were visible via Assets>Tokens when I loaded it in. However Unity crashed when I tried to open the module - I suspect because it couldn't find the tokens for the monsters since I was unable to delete the token links from the .xml file.

So, I opened the module in Classic; made a copy of a few monsters - removed the tokens by giving them another token which I knew to be in .png format and exported just those few monsters. The module then opend fine in Unity. So the solution for this file I think is you'll need to change the format of the .gif tokens to .png (you'll need some kind of graphics program to do that) and then re-allocate the new tokens to the NPC's and finally export to a module.

By the way the file is truly massive and I am surprised that with this file open in classic you don't get huge memory issues. This is a file which is shared by default so your players are able to access this and at this size again I am surprised that the clients don't crash as well. With nothing else open the file is using over 1Gb DM side and because it is full of tokens that will be eating a huge amount of memory client side.

So even for Classic the file has massive issues. I am truly astonished that you were able to play a game in Classic with this size of module shared with your players. Monster Manuals or Bestiaries are not something a DM would normally share with the players and the size of this one is staggering.

I also looked at the Anlira module and again this is a massive thing. Again it contains a huge number of tokens mostly in .gif format. The tokens are also all over the place; partly in the campaign folder; partly in tokens/host and partly in tokens/shared. Again the tokens can be seen from Assets>Tokens but only a very small number appear and again that'll be a formatting issue. A large part of the module contains images and most of those are fine in terms of format and size, however although the module appeared in Unity, when opened it threw an error related to a file which has an & (ampersand) in it's name. I changed the name of that file and removed the ampersand from the db.xml; zipped it all back up and the module then opens in Unity. So the problem with that module was just the one file '11- Widdershins - 22 Furrow & Vine'. Renaming that file to remove the ampersand and re-exporting should therefore solve the issue (at least as far as the non tokens bit goes - there are also some images in .gif or .png format that'll need to be changed to .jpg).

So most of your issues it would seem are with tokens being in the wrong format. Changing those to .png files should solve a lot of your issues. But I would also consider making the modules a lot smaller - especially anything shared with your players.

I didn't look at the other file but I'm going to assume that similar problems exist in that one too.

[Zacchaeus]

Ok, I couldn't resist looking at the Varisia file too. Man, you like big modules - this one is an absolutely incredible 1GB in size!

The problem is again the massive amount of tokens (most of which I think are duplicated in the other modules as well) which are all in .gif format. The bulk of the module is made up of the 2,000 images. Some of these are also in .gif format and there are also several maps in .png format and one of those is 11Mb in size, with several others exceeding 1Mb. I would certainly look to converting all of the maps to .jpg format.

I could not get this module to even appear initially in Unity but I did get it to show up eventually by first deleting all of the images with a .gif file extension and then I found several maps which had an ampersand (&) in their title. Once I had removed the ampersand from those maps in the .xml file and zipped it back up the module opened in Unity. So in order to get this one working; change the .gif images to .jpg; remove the ampersand from any image names and redo any tokens with .gif to .png and then re-export the module from your creation campaign. You should then be able to open the module in Unity.

Post script: I would recommend that you create a token module or modules - follow this video here (https://www.fantasygrounds.com/forums/showthread.php?43011-How-to-make-a-Token-Module&p=382389#post382389) to see how to do that. Once you have done so then you can allocate a token from this module to any NPCs in any campaign and those links will stay even if the token module is then closed. From what I saw every one of your modules all have basically the same tokens and that means that you are using way more memory than needed and the modules themselves are massive. Going forward you may want to consider map modules as well for your map images.

[dellanx]

Thank you for looking at this Zacchaeus.

I have tried to edit the Anlira and other mods. Changed the gif to png. I edited the db.xml file to change gif to png being careful not change some other text *gif* by matching the exact word. I edited out the "&", but could not get the new mod to load in Unity. Finally I tried to export a mod using the same instant that I exported module 1 - Part 3, i did not put the png tokens in. That module I called it "Transfer" did import into unity, but the 2300 NPCs crashed unity, while it is fine in FG classic. Other problem may be is the new module is looking for the Anlira and Varisa modules, and then things crash. Not sure, but I found references to thos modules in db.xml, but I can't load thise modules in unity. May I send you the edited Anlira module, so you can look at it and let me know where I went wrong?


I am not sure where I went wrong in the edits. The only data I can re-export has 2300 npc's. I wonder if there is any easy way of break the NPCs up into separate bite size modules, besides some heavy duty deleting etc.

Thanks for all the effort.

p.s. way back when I saw that Gif would work and Png was an option, how could I have known that Unity would not support Gif. LOL

[Zacchaeus]

Thank you for looking at this Zacchaeus.

I have tried to edit the Anlira and other mods. Changed the gif to png. I edited the db.xml file to change gif to png being careful not change some other text *gif* by matching the exact word. I edited out the "&", but could not get the new mod to load in Unity. Finally I tried to export a mod using the same instant that I exported module 1 - Part 3, i did not put the png tokens in. That module I called it "Transfer" did import into unity, but the 2300 NPCs crashed unity, while it is fine in FG classic.

I am not sure where I went wrong in the edits. The only data I can re-export has 2300 npc's. I wonder if there is any easy way of break the NPCs up into separate bite size modules, besides some heavy duty deleting etc.

Thanks for all the effort.

You can't just change the file extension - that does not change the file format. You will need to load each individual token into a paint package like GIMP or Photoshop or similar and export it in .png format. Or find some program that will bulk convert the tokens into .png format. Editing the xml will not help you to sort the issue at all; you will need to actually sort the tokens. As I noted above once you change the tokens from .gif to .png you will then need to relink those to the NPCs. And only once you have done all that can you export it again to a module.

Re-exporting the NPCs won't help either since they will still be linked to a token. The only way you can solve this is to change the tokens to the correct format or make or find new tokens and replace the ones currently on the NPCs before exporting (whilst remembering to delete all the existing ones).

[dellanx]

You can't just change the file extension - that does not change the file format. You will need to load each individual token into a paint package like GIMP or Photoshop or similar and export it in .png format. Or find some program that will bulk convert the tokens into .png format. Editing the xml will not help you to sort the issue at all; you will need to actually sort the tokens. As I noted above once you change the tokens from .gif to .png you will then need to relink those to the NPCs. And only once you have done all that can you export it again to a module.

Re-exporting the NPCs won't help either since they will still be linked to a token. The only way you can solve this is to change the tokens to the correct format or make or find new tokens and replace the ones currently on the NPCs before exporting (whilst remembering to delete all the existing ones).

I used "BulkImageConverter.exe" it converted GIF to PNG. Edited the db.xml accordingly, removed "&" from file and db.xml. All tokens in PNG format now. Still FGU does not see the module.

[Mortar]

PM me the module Dellanx

[dellanx]

This maybe to far gone, and it maybe easier to start fresh.

[dellanx]

PM me the module Dellanx

Will do. I am sending the old and the edited modules as well as "Transfer.mod" that i exported and it loads, but FGU is slow.

[dellanx]

Zacchaeus thanks, the 30980 idea is awesome, just tried in in FGU worked like a charm.

[Zacchaeus]

I used "BulkImageConverter.exe" it converted GIF to PNG. Edited the db.xml accordingly, removed "&" from file and db.xml. All tokens in PNG format now. Still FGU does not see the module.

Ok, I got this working. Two things. Make sure that you are converting all of the tokens - some of the tokens are buried 4 or 5 layers deep in the tokens folder. Make sure also that you are deleting all of the .gif files whilst you do the conversion leaving only the .png files. Secondly when you zip the file back up make sure that you zip up the contents of the folder and not the folder itself. That is select the tokens folder the definition.xml and the client.xml files and zip that up (oh, and don't change the name of the module).

The module takes a veritable age to open up but it will eventually open up and all the NPCs are there. If you still can't get it to work give me a shout and I'll upload the module that I have. (I'd do that now but my internet right now is as slow as treacle due to Christmas traffic).

[dellanx]

Ok, I got this working. Two things. Make sure that you are converting all of the tokens - some of the tokens are buried 4 or 5 layers deep in the tokens folder. Make sure also that you are deleting all of the .gif files whilst you do the conversion leaving only the .png files. Secondly when you zip the file back up make sure that you zip up the contents of the folder and not the folder itself. That is select the tokens folder the definition.xml and the client.xml files and zip that up (oh, and don't change the name of the module).

The module takes a veritable age to open up but it will eventually open up and all the NPCs are there. If you still can't get it to work give me a shout and I'll upload the module that I have. (I'd do that now but my internet right now is as slow as treacle due to Christmas traffic).

I was zipping the folder not the content. Neat, Thank you! Now need to try same to Varisia.mod.

[dellanx]

Tried to load Verisia, first try did not work.

[dellanx]

Tried to load Verisia, first try did not work.

I finally settled on three modules:

1 token module: Varisia Tokens

2 Modules for everything else: 1.0 Anlira.mod is 33.7 MB and 2.0 Anlira-Varisia.mod is 315 MB.

Is there anyway I could make a separate module just for images and refer both modules to it?

Thanks, for everything so far.

p.s. or could I refer 1.0 Anlira.mod to 2.0 Anlira-Varisia.mod for images? - It is on the ted size of sluggish, but everything did transfer. However it would be nice to improve things.

[Zacchaeus]

Yes, you can make a module out of anything including just images. You can link the images to anything and as long as the image module is open the links will work.

The way Unity works, as far as images are concerned, is you can access all of them via assets; and all of your images are available all of the time even if the module isn't open. You can drag any image from assets into your campaign at any time. So this means that for homebrew things you don't necessarily need to have every image in the actual module all of the time. You could just add images required for the next session during prep. I didn't look at the modules closely but do you need 2,000 images for that one campaign? Or did you just put all of your images into that one module?

The modules are the biggest things I have ever seen and I would certainly be looking to make them smaller. Perhaps splitting the campaign into chapters - if that is possible; you could for example have one module for general maps, history, and background information on the world. Another with part one of the story; another with part 2 etc. It all depends on how you organise things and how you run it, but I would not just keep adding more and more to the same module.

Memory in Unity is certainly not going to be the issue it is in Classic but nevertheless we can't just throw caution to the winds and stuff everything into one campaign.

[dellanx]

Yes, you can make a module out of anything including just images. You can link the images to anything and as long as the image module is open the links will work.

The way Unity works, as far as images are concerned, is you can access all of them via assets; and all of your images are available all of the time even if the module isn't open. You can drag any image from assets into your campaign at any time. So this means that for homebrew things you don't necessarily need to have every image in the actual module all of the time. You could just add images required for the next session during prep. I didn't look at the modules closely but do you need 2,000 images for that one campaign? Or did you just put all of your images into that one module?

The modules are the biggest things I have ever seen and I would certainly be looking to make them smaller. Perhaps splitting the campaign into chapters - if that is possible; you could for example have one module for general maps, history, and background information on the world. Another with part one of the story; another with part 2 etc. It all depends on how you organise things and how you run it, but I would not just keep adding more and more to the same module.

Memory in Unity is certainly not going to be the issue it is in Classic but nevertheless we can't just throw caution to the winds and stuff everything into one campaign.

Anlira has Story content that is not easy to replace, while 2.0 Anlira-Varisia has active game content. 2.0 Anlira-Varisa should have all images that is needed. Things are still a bit sluggish, but no errors, except Cgen does not work properly. Thanks

p.s. Sent the latest to you and Zacchaeus. Off to sleep. Now that I have edited code, things make more sense.

[dellanx]

If I am looking for the Images.mod for images in a Folder called "Images" would I do this?


<id-00122>
<image type="image">Images
<bitmap>E2. Leftover.jpg@Images</bitmap>
</image>
<name type="string">E2. Leftover</name>
</id-00122>


Thanks

[dellanx]

Ah got it:

<id-00054>
<image type="image">
<bitmap>Images/2.5 Seven Swords of Sin - Terana.jpg@Images</bitmap>
</image>
<name type="string">2.5 Seven Swords of Sin - Terana</name>
</id-00054>


30986

[Mortar]

I have finally had a chance to look at 2.0 Anlira-Varisia Final mod you pm'd. My comments are directed specifically at this module and ways to decrease its overall size, but they can be applied to any module. Note: I have never seen a
db.xml file this big before - both Notepad++ and Sublimetext3 have issues with this file on my desktop.

None of this is meant to belittle the work you have done, but to make your work and that module more efficient for you.

1. Consider removing the
<modifiers></modifiers> from the
db.xml. Its an empty section, and a waster of space. Yeah its only 193 characters, but in a module of this size every character counts. (The whole module has ~103 million characters in it.)

2. Are the feats, items, and spells going to be usable by the player's in your campaign? If so, maybe consider putting them into their own player module which is shared by default. You as the GM don't necessarily have to have that module loaded all the time, and that info isn't really needed in the 2.0 Anlira-Varisia mod.

3. Do the same thing with the Pre-generated characters (which I am guessing might be your PCs from the campaign). Chances are the only time those will used is when you first start up the campaign in FGU to continue playing it. Once the players have selected their characters you can close this module as it won't be needed. The way it is set-up right now the pre-gens will always be loaded into memory when 2.0 Anlira-Varisia is loaded. That would mean a copy in this module, plus a copy in the campaign database. More space wasted.

4. Story entries - there are a LOT of these - 2753 of these to be exact which means this is the area where you can shave a serious amount of space off the module.

a. Are all these story entries a combination of campaign history and the campaign's future? Anything that is campaign history, maybe consider putting those entries into a separate module?

b. Check your character encoding. Recent font updates in FGU means it will recognize characters in the
db.xml or client.xml from the following character sets: Basic Latin, Latin-1 Supplement, Latin Extended-A, and Latin Extended-B. Almost all the characters you have encoded can be replaced with plain text. This (https://www.utf8-chartable.de/unicode-utf8-table.pl) page will be helpful. Set both "display format for UTF-8 encoding" and "numerical HTML encoding of the Unicode character" to find the encoding you do need. En-dash is
& # 8211;, with the Em-dashes being
& # 8212;, they are both under the "General Punctuation" list on the page I linked.

c. While I understand why you used text tables and empty paragraph spacing in the entries, I wouldn't use them myself. No idea how much space that will save because I haven't attempted it yet.