Item6009: WYSIWYG corrupts embedded quotes that are escaped.

The following string (First instance "sticky", second instance not.)

PARAMETER="Text with "Escaped" quotes"

PARAMETER="Text with "Escaped" quotes" 

Looses the " after a pass through the TinyMCE editor. This string started out with an escaped quote, and will end up with a raw quote. If initially entered in TinyMCE, it survives the "First Save" and will be converted to a raw quote after the 2nd edit/save cycle. If it is pre-existing in the topic or entered in raw mode, it is lost on the first edit with TinyMCE.

This will break any plugin tags that have to pass in an escaped quote. The 2nd edit/save will result in an early close-quote and a truncated parameter. This will have a significant impact with migration to 4.2. Minor changes in one part of existing topics can result in broken rendering in other parts of a topic. I don't see how we can expect our users to find and surround these strings with <sticky tags. Migrating 10's of thousands of topics to find all these things that don't survive a pass throught the 4.2 default editor will be a significant challenge.

The documented WYSIWYG_EXCLUDE  paremeter is not set by default, and I don't understand the examples on how to set this. Does TinyMCE honor it?

-- TWiki:Main/GeorgeClark - 21 Sep 2008

Note that the same thing happens to the Ampersand variable. So if you try to use &amp;quote, the first pass changes the & amp; to a &, and then the next edit changes the & quote to ".

-- TWiki:Main.GeorgeClark - 21 Sep 2008

Issues related to handling of HTML entities are usually related to WysiwygPlugin (don't worry, I wouldn't have expected anyone to know that). Yes, WYSIWYG_EXLCUDE works. Can you be more specific about what is unclear in the documentation? "I didn't understand it" isn't much help, as I understand it perfectly, and therefore can't reproduce the problem

Is the PARAMETER= in the example above meant to part of a * Set statement? If so, there may be a bug. if it is not part of a Set, then it is behaving correctly.

-- CrawfordCurrie - 23 Sep 2008

Hi Crawford, the parameter is part of the ToolTipPlugin tag. TWiki variables delimit parameters by double-quote. The ToolTip JS libraries require comma-separated single-quoted strings. The plugin manages that conversion, but has to have any embedded quotes in a parameter string "escaped". similarly to complex searches with embedded HTML.

So %TAG{Parameter="string with escaped double-quotes"}% the escapes are lost.

Take a look at http://twiki.org/cgi-bin/view/Plugins/ToolTipPlugin?raw=on Specifically there are a couple of examples - the 3rd 4th and 5th TOOLTIP examples had to go inside sticky tags to survive.

I didn't test with SEARCH, but I'm sure we use a number of searches with embedded quot; escapes. I can't imagine having to go find every search with escaped quotes to add the sticky tags.

Regarding settings and the WYSIWYG docs, it shows the default content of the WYSIWYG_EXCLUDE in a verbatim list, but doesn't actually show a SET statement. I've hunted around for where that is set, and expanded the allvariables variable and it is not set. I don't see any example on how to go from the shown verbatim list to add the ToolTipPlugin as an escape. If I set it does it replace the default or add to it? I think the example needs a real example of the required "set" statement that could be uses as a starting point.

-- TWiki:Main.GeorgeClark - 24 Sep 2008

I'm not sure I agree that the editor removing escaped quotes and ampersand outside of set statement is desirable behavior. With includes and doing things like building content in one topic through regex expression searches against another topic, I don't believe that it is appropriate for the editor to assume that html "character entities" can be replace with their literal equivalents.

I believe that if the user enters directly any html character entity, it really ought to be preserved.

-- TWiki:Main.GeorgeClark - 24 Sep 2008

Perhaps. TML constructs - such as tags, set statements, and literal/verbatim blocks, are automatically "protected" from the editor/browser by enforcing strict escaping of odd characters. Outside of these constructs, the requirement is that HTML entities be converted back to characters in the native character set of the TWiki installation. This requirement was imposed by people who regularly switch (or think they may switch) between WYSIWYG and raw editing. Obviously it would be easiest to maintain the HTML entities, but that tends to make raw editing (and searches) difficult.

-- TWiki:Main.CrawfordCurrie - 01 Oct 2008

From my testing with the %TOOLTIP, the contents of that tag was definitely not protected. Is there something that has to be configured to add TOOLTIP to the list of protected tags, or is it a bug?

On DirectedGraphPlugin, I'd need to protect the contents between the <dot> and </dot> markup. Since this markup isn't a standard tag, I'm assuming I need to include instructions to add something to the WYSIWYG configuration, but that comes back to my being confused by the docs.

-- TWiki:Main.GeorgeClark - 01 Oct 2008

I can confirm Crawford's observations. It is extra work having to protect the special constructs in these cases but the alternative is that a lot of functionality related to what is encoded and what is not is breaking. We have been there. It created problems. We should not go back re-creating a big problem to solve a more special problem. The Wysiwyg has the features needed to protect more special constructs.

And the mix of Raw and Wysiwyg editing is very popular. There are users that only raw edit topics. But no user I know in my user community edits Wysiwyg only. They mix it all the time.

-- TWiki:Main.KennethLavrsen - 02 Oct 2008

And yet where I run into this problem is specifically when I mix raw and Wysiwyg editing. Okay... re-reading the documentation, I see I've been doing TML wrong for years. &quot; is incorrect. \" is the correct escape.

So I guess we can close this as a dumb user error.

-- TWiki:Main.GeorgeClark - 05 Oct 2008