Clean Up Text error

[tjlee]

I get the following error when using the 'Clean Up Text' context menu button in cases involving link references:


[ERROR] FormattedText SetValue XML Error: Unexpected end of file has occurred. The following elements are not closed: p. Line 1, position 173

Steps to reproduce:

Create a story.
Add a line of text.
Press enter twice so there's an empty line.
Drag/drop a record link below the empty line.
Activate 'Clean Up Text'


Looking at the text dumps from the story (I have debug tools that help me), it looks like a <p /> that represents the empty line is getting mutated to <p> with no closing </p> after invoking ImportUtilityManager.cleanUpText(). A potential fix might be to expand <p /> to <p></p> at the start of cleanUpText() and let the rest of the clean-up happen as before.

Edit: Same error occurs if the link reference is changed to a chat frame text block (Ctrl-3)

[Zacchaeus]

I'm not seeing this. Is the story an advanced or simple story? What ruleset? What kind of link are you dragging in? (Items, story, NPC etc).

Anything else other than the steps you provided above?

Edit: Wait. I read what you said again, so dropping the link into the second empty line you made does create the error. I was dragging into the logical line below the line of text.

[tjlee]

It’s a simple story in a CoreRPG only campaign.

I dragged the link for another story into the formatted text field. Another way to reproduce is to have a line of text before the link and then add a blank line in between, then trigger ’Clean Up Text’. When you remove the empty line, the error goes away. If you add the empty line again, the error reappears.

Edit: Here's what it looks like:

65936

[Moon Wizard]

Just pushed a fix to CoreRPG that should address this.

Regards,
JPG

[tjlee]

I appreciate the quick turn-around for the fix - verified with latest version of CoreRPG.

[tjlee]

When a formatted text field contains the "<" (less than) character, the Clean Up Text context button generates an error like the following:

[ERROR] FormattedText SetValue XML Error: Name cannot begin with the '<' character, hexadecimal value 0x3C. Line 1, position 5.

To reproduce, create a new Story and simply type a less than character into the story and then invoke clean up text.

A potential work-around is to change the encoding of &# 60; (no space) to &lt; at the beginning of ImportUtilityManager.cleanUpText and then change it back just before it returns the results. Perhaps there's a better fix.

[Moon Wizard]

This one isn't my code; so will have to look at it or have Doug review.

I don't believe the work-around will work, since Doug wrote to handle basic HTML and that would break it.

Regards,
JPG

[tjlee]

With the latest changes where Clean Up Text got split into Clean Up -> Text and HTML, this now manifests in Clean up HTML.

[tjlee]

With the latest changes where Clean Up Text got split into Clean Up -> Text and HTML, this now manifests in Clean up HTML.

Hmm....just rechecked...seems to be resolved....odd....

[tjlee]

Hmm....just rechecked...seems to be resolved....odd....

Here's a more specific test case for Clean Up HTML. To reproduce, Copy/paste the following into a new story and activate Clean Up HTML:


11. Others (<0.5%): Tieflings, dragonborn extremely rare

Note: it's the less than symbol within this context that causes the error. If you remove it, there's no error. Add it back in, the error re-surfaces.

[Moon Wizard]

I've forwarded to Doug who wrote that code.

Regards,
JPG

[ddavison]

The < is a special character in HTML so it will think that is an open tag. Are you trying to use the < as the data within an HTML element?

[tjlee]

The < is a special character in HTML so it will think that is an open tag. Are you trying to use the < as the data within an HTML element?

To reproduce the error, apply Clean Up HTML to the following text in a story:


11. Others (<0.5%): Tieflings, dragonborn extremely rare

As you can see, the < character used above isn’t part of an open tag and shouldn’t result in an error message in the console.

[ddavison]

What is the full input you are trying to use?

[tjlee]

What is the full input you are trying to use?

I isolated the text to just that one line which reproduces the error. If you remove the < character, the error goes away. If you put it back in, the error reappears. Any added text doesn’t affect the error.

[ddavison]

I think you are missing my point. An unescaped < character will be treated as a special character if you ask an HTML interpreter to process it. It generating an error is appropriate in this case and operating as expected. I'm trying to find out if you are trying to input a large section of HTML that includes stand-alone, unescaped characters in it so that I can provide better guidance if this is your intended usage.

[tjlee]

I think you are missing my point. An unescaped < character will be treated as a special character if you ask an HTML interpreter to process it. It generating an error is appropriate in this case and operating as expected. I'm trying to find out if you are trying to input a large section of HTML that includes stand-alone, unescaped characters in it so that I can provide better guidance if this is your intended usage.

Simply copy/paste that text into a new story and activate Clean Up HTML context menu button on it to reproduce the error. There’s no programming involved here.

However, I’ve observed that by the time Clean Up HTML calls the underlying function with the formatted text, it will be surrounded by begin and end tags as a paragraph and the < character will be encoded in &#n; notation.

[ddavison]

Why are you asking it to clean up HTML if you are not supplying it HTML?

[tjlee]

I'm merely testing the as-built behavior - should errors spew to console when the end user invokes an available option?

Incidentally, the same error is reproduced (once again) by having a single character story of just "<".

Additionally, at the programming level, the same text is treated properly if I do this:


function myCleanUpHTML(sText)

sText = sText:gsub("&#60;", "&lt;");
sText = ImportUtilityManager.cleanUpHTML(sText);
sText = sText:gsub("&lt;", "&#60;");
return sText;
end

I'm basically translating the < encoded as &#60; to &lt; which ImportUtilityManager.cleanUpHTML() likely ignores?

If you want to test programmatically, try giving the function this input string:


<p>&#60;</p>

That's valid HTML isn't it?

[tjlee]

Actually, what causes the error is when the text having gone through cleanUpHTML is set back into the FormattedText via setValue which results in this error:


[ERROR] FormattedText SetValue XML Error: Name cannot begin with the '<' character, hexadecimal value 0x3C. Line 1, position 5.

[tjlee]

Which is caused by doing this in template_common.xml:


setValue(ImportUtilityManager.cleanUpHTML(getValue ()));

If you print out the text before it is set back into the formatted text field, you'll see improper HTML:


<p><</p>

Which means cleanUpHTML is likely not properly representing the data when it returns (should re-encode any special characters).

Sorry for the spam - just debugging this in real-time to give you as much info as possible.

[Moon Wizard]

The reason that Doug broke up to two input clean up methods is because Text and HTML are two different formats, which is what caused the problem you originally reported.

By having them separate, the system now assumes that if you choose Clean Up HTML, then you are inputting valid HTML, which the input you provided is not technically valid HTML, which is why the Microsoft parsing engine is rejecting that input.

To be specific, if you want to input that text and use the Clean Up HTML option, it will need to be valid HTML, which means the < would have to be escaped as &lt; or the hex code.

Regards,
JPG

[tjlee]

Let’s put it this way:

Valid HTML goes into the cleanUpHTML function (lua code) and invalid HTML comes out when the data portion includes a legitimate < character encoded as &#60; because it decodes the encoding as part of its processing and never re-encodes it when the cleaned up HTML is returned.

The downstream effect is that the now invalid HTML when set back into the formatted text field is rejected as invalid with a console message which is done in CoreRPG as I’ve already identified.