• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7848 for generic doc work for TWiki-6.1.1. Use View topic Item7851 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

Item3048: Documented that CGI version 2.89 causes character entities to not be escaped in Raw View

Item Form Data

AppliesTo: Component: Priority: CurrentState: WaitingFor: TargetRelease ReleasedIn
Engine Documentation Normal Closed   patch 4.2.1

Edit Form Data

Reported By:
Applies To:
Current State:
Waiting For:
Target Release:
Released In:


Type character entities such as | < or anything >. Looking at "Raw View" shows the rendered entities instead of raw text. Problem if one wants to recover an earlier version with copy & paste.

-- PTh

Hmm. The restore of early version or copy raw from other topic is a must. Increasing to requirement.

Was this also a bug in 4.0.4 or is this a new bug? KJL

Just checked, this bug is also in 4.0.2.

-- PTh

See http://develop.twiki.org/~twiki4/cgi-bin/view/LitterTray/RawMeat?raw=on and http://develop.twiki.org/~twiki4/cgi-bin/view/LitterTray/RawMeat?rev=1&raw=on- there doesn't seem to be any problem....? Can you narrow down on where the unexpected behaviour happens, and provide a testcase please?


OK, your LitterTray.RawMeat does work properly in view raw:

Examples: & | < >

I created exactly the same topic on my svn MAIN 11883, and a view raw shows this:

Examples: & | < >


% uname -a
Linux thoeny.org 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:59:02 EST 2004 i686 i686 i386 GNU/Linux

% perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
    osname=linux, osvers=2.4.21-25.elsmp, archname=i386-linux-thread-multi
    uname='linux porky.build.redhat.com 2.4.21-25.elsmp #1 smp fri nov 12 21:34:51 est 2004 i686 i686 i386 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -march=i386 -mcpu=i686 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Dotherlibdirs=/usr/lib/perl5/5.8.0 -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -g -pipe -march=i386 -mcpu=i686',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='3.2.3 20030502 (Red Hat Linux 3.2.3-54)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil
    perllibs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl):
  Locally applied patches:
  Built under linux
  Compiled at Dec 13 2005 16:27:05

I disabled all non-standard plugins. Content of LocalSite.cfg:

$TWiki::cfg{DefaultUrlHost} = 'http://*****';
$TWiki::cfg{ScriptUrlPath} = '/*****';
$TWiki::cfg{PubUrlPath} = '/pub';
$TWiki::cfg{PubDir} = '/*****/svn/MAIN/pub';
$TWiki::cfg{DataDir} = '/*****/svn/MAIN/data';
$TWiki::cfg{LogDir} = $cfg{DataDir};
$TWiki::cfg{UseClientSessions} = 0;
$TWiki::cfg{AntiSpam}{HideUserDetails} = 0;
$TWiki::cfg{Password} = 'DjVdtm2FA3IzM';
$TWiki::cfg{Site}{Lang} = 'en';
$TWiki::cfg{LocalesDir} = '/*****/svn/MAIN/locale';
$TWiki::cfg{Site}{FullLang} = 'en-us';
$TWiki::cfg{TemplateDir} = '/*****/svn/MAIN/templates';
$TWiki::cfg{Site}{CharSet} = 'iso-8859-15';
$TWiki::cfg{Plugins}{WysiwygPlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{RenderListPlugin}{Enabled} = 0;
$TWiki::cfg{Plugins}{TagMePlugin}{Enabled} = 0;
$TWiki::cfg{Plugins}{PublishWebPlugin}{Enabled} = 0;
$TWiki::cfg{Plugins}{BlackListPlugin}{Enabled} = 0;
$TWiki::cfg{Plugins}{EditSyntaxPlugin}{Enabled} = 0;
$TWiki::cfg{Plugins}{StopWikiWordLinkPlugin}{Enabled} = 0;
$TWiki::cfg{EnableHierarchicalWebs} = 1;
$TWiki::cfg{ScriptSuffix} = '';
$TWiki::cfg{OS} = 'UNIX';
$TWiki::cfg{DetailedOS} = 'linux';
$TWiki::cfg{Sessions}{Dir} = '/tmp';
$TWiki::cfg{Sessions}{ExpireAfter} = 21600;
$TWiki::cfg{Sessions}{IDsInURLs} = 0;
$TWiki::cfg{Sessions}{UseIPMatching} = 1;
$TWiki::cfg{Sessions}{MapIP2SID} = 0;
$TWiki::cfg{LoginManager} = 'TWiki::Client::ApacheLogin';
$TWiki::cfg{DefaultUserLogin} = 'guest';
$TWiki::cfg{DefaultUserWikiName} = 'TWikiGuest';
$TWiki::cfg{AdminUserWikiName} = 'TWikiAdminGroup';
$TWiki::cfg{SuperAdminGroup} = 'TWikiAdminGroup';
$TWiki::cfg{UsersTopicName} = 'TWikiUsers';
$TWiki::cfg{AuthScripts} = 'attach,edit,manage,rename,save,upload,viewauth,rdiffauth';
$TWiki::cfg{AuthRealm} = 'Enter your TWiki.LoginName. (Typically First name and last name, no space, no dots, capitalized, e.g. !JohnSmith, unless you chose otherwise). Visit TWiki.TWikiRegistration if you do not have one.';
$TWiki::cfg{PasswordManager} = 'TWiki::Users::HtPasswdUser';
$TWiki::cfg{MinPasswordLength} = 1;
$TWiki::cfg{Htpasswd}{FileName} = '/*****/svn/MAIN/data/.htpasswd';
$TWiki::cfg{Htpasswd}{Encoding} = 'crypt';
$TWiki::cfg{UserMappingManager} = 'TWiki::Users::TWikiUserMapping';
$TWiki::cfg{SafeEnvPath} = '/bin:/usr/bin';
$TWiki::cfg{UploadFilter} = qr/(?-xism:(?-xism:(?-xism:(?-xism:^(\.htaccess|.*\.(?i)(?:php[0-9s]?(\..*)?|[sp]htm[l]?(\..*)?|pl|py|cgi))$))))/;
$TWiki::cfg{NameFilter} = qr/(?-xism:(?-xism:(?-xism:(?-xism:[\s\*?~^\$@%`"'&;|<>\x00-\x1f]))))/;
$TWiki::cfg{AntiSpam}{EmailPadding} = '';
$TWiki::cfg{ConfigurationLogName} = '/*****/svn/MAIN/data/configurationlog.txt';
$TWiki::cfg{DebugFileName} = '/*****/svn/MAIN/data/debug.txt';
$TWiki::cfg{WarningFileName} = '/*****/svn/MAIN/data/warn%DATE%.txt';
$TWiki::cfg{LogFileName} = '/*****/svn/MAIN/data/log%DATE%.txt';
$TWiki::cfg{Languages}{de}{Enabled} = 1;
$TWiki::cfg{Languages}{es}{Enabled} = 1;
$TWiki::cfg{Languages}{fr}{Enabled} = 1;
$TWiki::cfg{Languages}{it}{Enabled} = 1;
$TWiki::cfg{Languages}{nl}{Enabled} = 1;
$TWiki::cfg{Languages}{pl}{Enabled} = 1;
$TWiki::cfg{Languages}{pt}{Enabled} = 1;
$TWiki::cfg{Languages}{sv}{Enabled} = 1;
$TWiki::cfg{Languages}{'zh-cn'}{Enabled} = 1;
$TWiki::cfg{Languages}{'zh-tw'}{Enabled} = 1;
$TWiki::cfg{DisplayTimeValues} = 'gmtime';
$TWiki::cfg{Site}{Locale} = 'en_US.ISO-8859-1';
$TWiki::cfg{UpperNational} = '';
$TWiki::cfg{LowerNational} = '';
$TWiki::cfg{StoreImpl} = 'RcsWrap';
$TWiki::cfg{RCS}{ExtOption} = '';
$TWiki::cfg{RCS}{dirPermission} = 509;
$TWiki::cfg{RCS}{filePermission} = 420;
$TWiki::cfg{RCS}{asciiFileSuffixes} = qr/(?-xism:(?-xism:(?-xism:(?-xism:\.(txt|html|xml|pl)$))))/;
$TWiki::cfg{RCS}{EgrepCmd} = '/bin/egrep %CS{|-i}% %DET{|-l}% -H -- %TOKEN|U% %FILES|F%';
$TWiki::cfg{RCS}{FgrepCmd} = '/bin/fgrep %CS{|-i}% %DET{|-l}% -H -- %TOKEN|U% %FILES|F%';
$TWiki::cfg{RCS}{initBinaryCmd} = '/usr/bin/rcs  -i -t-none -kb %FILENAME|F%';
$TWiki::cfg{RCS}{initTextCmd} = '/usr/bin/rcs  -i -t-none -ko %FILENAME|F%';
$TWiki::cfg{RCS}{tmpBinaryCmd} = '/usr/bin/rcs  -kb %FILENAME|F%';
$TWiki::cfg{RCS}{ciCmd} = '/usr/bin/ci  -m%COMMENT|U% -t-none -w%USERNAME|S% -u %FILENAME|F%';
$TWiki::cfg{RCS}{ciDateCmd} = '/usr/bin/ci  -m%COMMENT|U% -t-none -d%DATE|D% -u -w%USERNAME|S% %FILENAME|F%';
$TWiki::cfg{RCS}{coCmd} = '/usr/bin/co  -p%REVISION|N% -ko %FILENAME|F%';
$TWiki::cfg{RCS}{histCmd} = '/usr/bin/rlog  -h %FILENAME|F%';
$TWiki::cfg{RCS}{infoCmd} = '/usr/bin/rlog  -r%REVISION|N% %FILENAME|F%';
$TWiki::cfg{RCS}{rlogDateCmd} = '/usr/bin/rlog  -d%DATE|D% %FILENAME|F%';
$TWiki::cfg{RCS}{diffCmd} = '/usr/bin/rcsdiff  -q -w -B -r%REVISION1|N% -r%REVISION2|N% -ko --unified=%CONTEXT|N% %FILENAME|F%';
$TWiki::cfg{RCS}{lockCmd} = '/usr/bin/rcs  -l %FILENAME|F%';
$TWiki::cfg{RCS}{unlockCmd} = '/usr/bin/rcs  -u %FILENAME|F%';
$TWiki::cfg{RCS}{breaklockCmd} = '/usr/bin/rcs  -u -M %FILENAME|F%';
$TWiki::cfg{RCS}{delRevCmd} = '/usr/bin/rcs  -o%REVISION|N% %FILENAME|F%';
$TWiki::cfg{RCS}{WorkAreaDir} = '/*****/svn/MAIN/pub/_work_areas';
$TWiki::cfg{SystemWebName} = 'TWiki';
$TWiki::cfg{TrashWebName} = 'Trash';
$TWiki::cfg{UsersWebName} = 'Main';
$TWiki::cfg{WebMasterEmail} = 'peter@thoeny.org';
$TWiki::cfg{WebMasterName} = 'TWiki Administrator';
$TWiki::cfg{MailProgram} = '/usr/sbin/sendmail -t -oi -oeq';
$TWiki::cfg{SMTP}{MAILHOST} = '';
$TWiki::cfg{SMTP}{SENDERHOST} = '';
$TWiki::cfg{SMTP}{Username} = '';
$TWiki::cfg{SMTP}{Password} = '';
$TWiki::cfg{NotifyTopicName} = 'WebNotify';
$TWiki::cfg{SMTP}{Debug} = 0;
$TWiki::cfg{PROXY}{HOST} = '';
$TWiki::cfg{PROXY}{PORT} = '';
$TWiki::cfg{Stats}{TopViews} = 10;
$TWiki::cfg{Stats}{TopContrib} = 10;
$TWiki::cfg{Stats}{TopicName} = 'WebStatistics';
$TWiki::cfg{LinkProtocolPattern} = '(file|ftp|gopher|https|http|irc|mailto|news|nntp|telnet)';
$TWiki::cfg{SiteWebTopicName} = '';
$TWiki::cfg{SitePrefsTopicName} = 'TWikiPreferences';
$TWiki::cfg{LocalSitePreferences} = 'Main.TWikiPreferences';
$TWiki::cfg{HomeTopicName} = 'WebHome';
$TWiki::cfg{WebPrefsTopicName} = 'WebPreferences';
$TWiki::cfg{NumberOfRevisions} = 4;
$TWiki::cfg{ReplaceIfEditedAgainWithin} = 3600;
$TWiki::cfg{LeaseLength} = 3600;
$TWiki::cfg{LeaseLengthLessForceful} = 3600;
$TWiki::cfg{MimeTypesFileName} = '/*****/svn/MAIN/data/mime.types';
$TWiki::cfg{RegistrationApprovals} = '/*****/svn/MAIN/data/RegistrationApprovals';
$TWiki::cfg{Plugins}{CommentPlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{EditTablePlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{InterwikiPlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{PreferencesPlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{SlideShowPlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{SmiliesPlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{SpreadSheetPlugin}{Enabled} = 1;
$TWiki::cfg{Plugins}{TablePlugin}{Enabled} = 1;
$TWiki::cfg{PluginsOrder} = 'SpreadSheetPlugin';
$TWiki::cfg{PassthroughDir} = '/tmp';
$TWiki::cfg{LoginNameFilterIn} = qr/(?-xism:(?-xism:(?-xism:^[^\s\*?~^\$@%`"'&;|<>\x00-\x1f]+$)))/;
$TWiki::cfg{MapUserToWikiName} = 1;
$TWiki::cfg{Register}{HidePasswd} = 1;
$TWiki::cfg{Register}{NeedVerification} = 1;
$TWiki::cfg{DenyDotDotInclude} = 1;
$TWiki::cfg{AllowInlineScript} = 1;
$TWiki::cfg{AntiSpam}{RobotsAreWelcome} = 1;
$TWiki::cfg{Log}{view} = 1;
$TWiki::cfg{Log}{search} = 1;
$TWiki::cfg{Log}{changes} = 1;
$TWiki::cfg{Log}{rdiff} = 1;
$TWiki::cfg{Log}{edit} = 1;
$TWiki::cfg{Log}{save} = 1;
$TWiki::cfg{Log}{upload} = 1;
$TWiki::cfg{Log}{attach} = 1;
$TWiki::cfg{Log}{rename} = 1;
$TWiki::cfg{Log}{register} = 1;
$TWiki::cfg{Languages}{da}{Enabled} = 1;
$TWiki::cfg{Site}{LocaleRegexes} = 1;
$TWiki::cfg{PluralToSingular} = 1;
$TWiki::cfg{AutoAttachPubFiles} = 1;
$TWiki::cfg{RemoveImgInMailnotify} = 1;
$TWiki::cfg{Plugins}{TwistyPlugin}{Enabled} = 1;
$TWiki::cfg{PublishWebPlugin}{PublishPath} = "../../../httpdocs";
$TWiki::cfg{PublishWebPlugin}{AttachPath}  = "_publish";

PS: Item3078 is another weird behaviour I cannot reproduce here on develop.twiki.org.

-- PTh

I tried the small test topic on both my latest SVN TWiki and on my 4.0.5 and it works as it should on both so it is related to your specific settings.

-- KJL

Works for me as well. This would call for a debugger session, or at least a run from the command line. Maybe an ancient CGI.pm is on Perl's @INC path before the "good" one? Maybe there's some post-processing on Peter's web server before it hits the browser?

-- haj

Switched to Waiting for Feedback from Peter, because I can't reproduce it either with of Peter's LocalSite settings. Possibly something to do with the browser?


My machine has CGI version 2.89.

I tested it on FF and IE, same issue. Seems not to depend on browser used.

-- PTh

Ah, it might be that, then, Some earlier (per 3.0) versions of CGI had problems with character encodings. if you can, please try upgrading your CGI. if the problem goes away, then we know that was the cause and can document the requirement for CGI >= 3.0 (actually IIRC it's already CGI >= 3.09 because of the POST flushing issue, though that may not be reflected in the docs)


My Centos 4.4 (the latest version and equivelent to latest version of RedHat Enterprise) has CGI of version 3.05 so I sure hope we are not having a 3.09 requirement now.

The 3.05 does not have the problem in this bug report. It is showing the entities correctly.

So maybe the right requirement that should be documented is CGI >= 3.0

Peter - I think we wait for your confirmation to know for sure that the bug in CGI is the root cause.


I can confirm that upgrading to CGI 3.25 solves the issue.

I suggest to review the code if there is a missing escape that makes this work on all CGI versions. It is possible that recent CGI modules do extra work to fix incorrect input. Please note that Cairo escapes content properly with CGI version 2.89.

-- PTh

I restored the HTML entity escapes for raw view that were taken out from Cairo code. Content is now properly escaped in all CGI module versions.

-- PTh

This can't be the solution as now twiki markup, e.g. % and the like are html entities now in raw mode. See http://develop.twiki.org/~twiki4/cgi-bin/view/LitterTray/TestTopic3?raw=on

Oh, and the other tests above are encoded twice now: &amp;amp;

-- MD

I can confirm that now the raw view is no longer working.

On my test test servers TWiki's raw view is totally goofed up.

This latest change has to be reverted. There is no way we can release TWiki with this new bug.

Also see here how totally goofed up it is: http://merlin.lavrsen.dk/twiki/bin/view/Myweb/FormMetaTest?raw=on


That code seems to have been reverted.

Cairo didn't have the bug because Cairo did all it's own entity conversions, in little fragments of mostly duplicated (and often inconsistent) code peppered throughout the codebase. CGI claims to provide consistent support for encoding, so it makes more sense to use that. If CGI is broken, let's help the authors of CGI get it right, instead of just ignoring them.

Note that some versions of CGI were acknowledged by the CGI authors to be SNAFU. Entity encoding was one bug; there was another around that time that was more fatal - I forget the details, but I think it was in 3.09. Without a rigorous testing procedure it is impossible to be sure that TWiki works with any specific version of anything. For example, AFAIK we still persist in claiming perl 5.006 support, though the days when that was even cursorily tested are long gone.

I recommend checking the CGI version in configure. I know 3.17 works, because I use it. Any earlier bids?


I set the requirement to 3.05. Closing.


Not sure what is the best solution to fix this bug, but we can't simply raise the required environment at will. By rasing requirement to CGI to 3.05 we will inevitably lose more evaluators because they get stuck in the installation process.


  • Ciaro handles entities properly in edit and view raw on the cgi versions tested
  • Edinburgh handles entities properly in edit on the cgi versions tested
  • Edinburgh does not handles entities properly in view raw on the cgi versions tested

This bug needs more research

-- PTh

I reverted the change on the hard requirement on CGI 3.05. We need to fix this bug without breaking existing installs, and without raising the requirements.

-- PTh

I also reverted my entity escape fix from a few days ago; the fix does not work on all CGI versions. So we are now back to the original state with bug shown when using CGI version 2.89.

-- PTh

I believe this is a documentation fix; it needs to be documented that it doesn't work with CGI 2.89. As Peter experienced, fixing it in the core is not an option.


No, I believe, fixing the core is the option. It works in Cairo, so it should be possible to fix in TWiki 4.x. We can't arbitrarily raise the required env.

-- PTh

I'm all for eliminating as many as our own/internal conversions as possible, and instead ask for installations to update their perl modules as they are bugfixed. My best guess would be that Cairo doesn't work with the whole range of CGI.pm versions either.

An installation is not dependent on the systemwide perl modules, local modules are easily put in front of the search path (i.e. CpanContrib).

-- SteffenPoulsen - 22 Dec 2007

I have chosen to do the documentation fix suggested.

-- TWiki:Main.KennethLavrsen - 29 Jul 2008

Summary Documented that CGI version 2.89 causes character entities to not be escaped in Raw View
ReportedBy TWiki:Main.PeterThoeny
Codebase ~twiki4
SVN Range TWiki-4.1, Sun, 22 Oct 2006, build 11793
AppliesTo Engine
Component Documentation
Priority Normal
CurrentState Closed

Checkins TWikirev:12394 TWikirev:12413 TWikirev:12414 TWikirev:12415 TWikirev:12416 TWikirev:17196 TWikirev:17197
TargetRelease patch
ReleasedIn 4.2.1
Edit | Attach | Watch | Print version | History: r36 < r35 < r34 < r33 < r32 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r36 - 2008-08-04 - KennethLavrsen
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback