• Do not register here on develop.twiki.org, login with your twiki.org account.
• Use View topic Item7848 for generic doc work for TWiki-6.1.1. Use View topic Item7851 for doc work on extensions that are not part of a release. More... Close
• Anything you create or change in standard webs (Main, TWiki, Sandbox etc) will be automatically reverted on every SVN update.
Does this site look broken?. Use the LitterTray web for test cases.

On each and every page view my warn200511.txt file is currently being flooded with these warnings.

| 23 Nov 2005 - 09:10 | AliasPlugin defines deprecated endRenderingHandler
| 23 Nov 2005 - 09:10 | AliasPlugin defines deprecated outsidePREHandler
| 23 Nov 2005 - 09:10 | BlackListPlugin defines deprecated endRenderingHandler
| 23 Nov 2005 - 09:10 | WysiwygPlugin defines deprecated endRenderingHandler
| 23 Nov 2005 - 09:10 | WysiwygPlugin defines deprecated writeHeaderHandler

Are these development messages that will eventually go? My warn file is being totally flooded with these and they must take extra time for Dakar which is already slow.

I am sure I will get even more of them if I enable more plugins.

-- TWiki:Main.KennethLavrsen

I find this is too radical. Lets remove those warning messages for the release where it is deprecated. Better to spread that out over two releases, e.g. start warn about Dakar deprecated functions in Edinburgh.

-- PTh

Dakar is the release where these function calls have been deprecated. they will disappear in a future version, right? i think part of the purpose was to cause (a tiny amount of) pain, in order to prod developers to update their plugins.

maybe i didn't understand what you meant. what does "remove those warning messages for the release where it is deprecated" mean?

i'm not really arguing one way or the other yet, just providing more information.

-- WN

  • Release X: Deprecate function Y, do not generate any warning
  • Release X + 1: Generate warnings if function Y is used
  • Release X + n: Retire function Y

Do not make it painful for the administrators, it is not their fault that we deprecate stuff. Ask the developers to fix the Plugins (conditionally, so that it runs on Cairo and Dakar (many companies are one or two releases behind))

-- PTh

These are developer warnings, not sysadmin warnings. Either have a flag to have these log only in developer mode (and for the specific plugin the admin is developing) or email each warning to the plugin developer every time one is generated wink

-- MC


PLUGINS AUTHORS NEED FEEDBACK. I considered all these options. It is a very bad idea to pretend that "everything is fine", and not highlight these problems. That's how we ended up with huge numbers of unloved, uncared-for plugins, rotting in the stocks.

These are not developer warnings; they are an essential aid to helping admins determine the quality of the plugins they are using. My idea was to increase the pester factor, to get admins to directly encourage plugins authors to update their plugins. By informing the admin, you give them the information they need to pester the plugin author directly, or even better, the opportunity to contribute a fix back to the community. *The plugin author gets warm fuzzies from knowing that people use their plugin, and care enough to want to get it fixed.

  • The admins get warm fuzzies from knowing they have contributed.
  • Peter gets warm fuzzies from knowing he isn't solely responsible for updating the handlers in 170 plugins.

An email only works if (1) you know the address to send it to and (2) someone on the other end cares.

Having said that, I am aware that some old-style handlers have to be kept to handle Cairo compatibility. I am keen to filter out the warnings in this case - for example, the WysiwygPlugin needs to defined deprecated endRenderingHandler for Cairo, but uses postRenderingHandler with Dakar.

Idea:

use vars qw( %compatibility );
$compatibility{endRenderingHandler} = 'Cairo';
If the handler has an entry in %compatibility, the plugin author is acknowedging that they know about the deprecation, but are ignoring it for compatibility reasons.

CC


We are pestering the 100000s of users and 1000s of administrators to send a message to 10 plugin authors. That is not a smart way to do it.

Having maybe 10 plugins adding these messages on each and every page view is not at all smart. Your idea does not address this issue. You will still be pestering and slowing down TWiki for all the users to send a message that you might as well just plainly write on TWiki.org on a plugin message topic and in the release notes in a INFORMATION FOR PLUGIN AUTHORS section.

And remember

  • Dakar is still too slow. Each and every plugin writing messages to a log is for sure adding to the load.
  • You drown the real warnings in what will be gigabytes of garbage log messages on a production server.
  • You waste disk space. This can be a huge problem when you are on a hosted service.

-- TWiki:Main.KennethLavrsen


We are pestering the 100000s of users and 1000s of administrators to send a message to 10 plugin authors - yep - it's called direct action, and it's how democracies have functioned for many, many years.

Proposal: (doc in TWikiPlugins)

Handling deprecated functions

From time-to-time, the TWiki developers will add new functions to the interface (either to FuncDotPm, or new handlers). Sometimes these improvements mean that old functions have to be deprecated to keep the interface manageable.

When this happens, the deprecated functions will be supported in the interface for at least one more TWiki release. When a plugin uses (or defines) the deprecated function, a warning will be generated in the TWiki warnings log. Admins who see these warnings in their logs should contact the plugin author for an updated version of the plugin.

Updated plugins may still need to define deprecated handlers for compatibility with old TWiki versions. In this case, the plugin package that defines old handlers can set a TWikiCompatibility global hash to prevent the warnings from being issued. This hash maps from the handler name to the TWiki::Plugins version in which the handler was first deprecated. For example, if we need to define the endRenderingHandler for compatibility with TWiki::Plugins versions before 1.1, we would add this to the plugin:

package SinkPlugin;
use vars qw( %TWikiCompatibility );
$TWikiCompatibility{endRenderingHandler} = 1.1;
If the currently-running TWiki version is 1.1 or later, then the handler will not be called and the warning will not be issued.

There is no way to prevent warnings from being issued when deprecated functions in the FuncDotPm interface are called. In this case, plugins authors should recode their plugins as described in the FuncDotPm documentation of the deprecated function.

CC

One of TWiki's greatest assets is the big number of Plugins. TWiki is a platform, not unlike an OS. The APIs can evolve, but please please keep it compatible. We need to take a slow gear with changing the API. An API is a promise to developers. If we violate that, developers will move on to another system. Not good for TWiki.

So, no warnings in Dakar for functions declared as deprecated, and retire them in, say, 2055. Well, earlier, but lets go slow.

-- PTh

No, I'm sorry, I cannot agree. If releases were a month apart, I might. But they are not - historically, they are a year apart.

APIs remain compatible, of course. That is what "deprecated" means - that the function is still there, it's just not to be used with new code. By giving people a horizon we are saying the deprecated function will stay in TWiki for at least one more release. We are not saying it will be removed after that, just that they are only only guaranteed until then. Slavishly keeping every function forever is totally impractical, and we need a way to phase them out when they get mouldy. We cannot allow the FuncDotPm interface to overconstrain the core.

(example: the line-by-line handlers, such as insidePreHandler, prevent many optimisations of the rendering loop. And they are very, very, very rarely used.)

There is of course a balance. Too much change is bad. Too little change, which you seem to be advocating, is just as bad. We need to strike a chord, and find the right amount of change that keeps people engaged and interested.

Given that the APIs are going to change - face it, the TWiki project will simply wither away (again) and die if they don't - then plugins authors need to be driven to update their plugins. Trying to over-protect plugins authors just causes code rot, as we have seen. Worse, "going slow" just makes people think the project is dead (or dying), and makes them drift away. This is an open-source project. So leverage the community to be more self-informative.

So, warnings in Dakar for deprecated functions and unguarded handlers, and retire them in Funfuti. Let's enable a more interactive community, and change up a gear, not down!

BTW the analogy to an OS just doesn't hold. No OS offers only one restricted API to it's functions. OS's are built in many layers, and allow programmers to tap in at all levels. And the APIs change regularly.

CC

SVN 7631 has the "escape clause" described above coded in, so Wysiwyg will not generate any more warnings.

Complete list of plugins that (1) compile and initialise and (2) declare deprecated handlers (and will therefore generate warnings):

24 Nov 2005 - 12:34 AliasPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 AliasPlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 BlackListPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 ChildTopicTemplatePlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 EditInTablePlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 EncryptedPagesPlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 ExplicitNumberingPlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 ExplicitNumberingPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 FileListPlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 FileListPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 FormPivotPlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 LaTeXToMathMLPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 LaTeXToMathMLPlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 LaTeXToMathMLPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 MultiEditPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 MultiEditPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 MultiLangPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 NatSkinPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 NavbarPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 RandomQuotePlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 RandomQuotePlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 RecursiveRenderPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 RecursiveRenderPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 RedDotPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 SectionalEditPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 StylePlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 StylePlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 SyntaxHighlightingPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 TigerSkinPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 TocPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 TodaysVisitorsPlugin defines deprecated outsidePREHandler
24 Nov 2005 - 12:34 TodaysVisitorsPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 ToolTipPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 TypographyPlugin defines deprecated endRenderingHandler
24 Nov 2005 - 12:34 TypographyPlugin defines deprecated startRenderingHandler
24 Nov 2005 - 12:34 XmlXslPlugin defines deprecated endRenderingHandler
CC
Duh!
More code to make the decision, and code that isn't going to be used but still gets compiled even when its never going to be executed. (I'll leave out the "more code, more to debug, more to go wrong" issue for the moment.)

We are clear on the point that the big slowdown with TWiki is compiling the code? So why compile code we're not going to use? I don't get it?

Some of the better (for various values of good) plugins already come in two parts, the one that does the heavy lifting ony dragged in when actually needed. (Mind you some of them could use more granuality.)

There's another way of looking at "backward compatibility" and that's recidivism. Right now, I see some comment here that smell of recidivism. Its as if the people who had an investment, techncial, emotional, in Bejing and Cairo won't let go, and further, want to drag the rest of us back. They may talk about advacnement, that's the noun, but where's the verb?

There are a a few other ways this can be approached. A draconian one is simply to insisit that the DEVELOP branch is for Dakar (at the moment) and simply don't put Cairo or Cairo compatible code in there. Period. Remove all these 'depreciated' functions. They are not there, they don't drag the compiler down, ther's no need to make if-the-else tests about what is running.

A less draconian approach would be to have an installed that optionally removed the Cairo compatible code at install time.

The bottom line is this: Dakar is not Cairo, it does thing differently. If we are to move forward we have to let go of the past. The sooner we let go the the sooner we can move forward. And the faster we can move forward.

But of course some people seem to be against progress and want to slow it down. Sorry, I'm not sympathetic when it comes to some key issues, and Kenneth raises an important one - performace.

-- AJA

I think the discussion is moving away from the original bug I reported.

I reported that adding a warning line in a log file for each deprecated function - from each plugin using them - for each page viewed as a problem. Easily 5-15 lines of text for each and every page view.

If I take the traffic from our Motorola server (number of read topics) and multiply by the number of bytes written into the warning log per plugin I will end up with 1-2 GB of warnings about deprecated functions PER MONTH. Someone on a popular TWiki site and a 100 MB hosted service will need to cleanout the warning log file almost daily to avoid running out of space.

My current Cairo warn file is 178 kilobytes after 14 months of production time. I actually see errors in it and look at it occationally and can react on them. The warn file would be 1.2 GB of noise if it had been Dakar on my Motion TWiki. It is one of the important tools we have to watch out for hackers trying to break in.

And you are writing the message to the poor TWiki administrators who cannot do anything about it and not to the plugin authors.

And 2nd - writing to a log file for each plugin loaded is for sure adding load and slowing down TWiki.

And I agree with AJA that adding more code to decide when to write the warnings is also slowing down TWiki. It can be OK to have such code during development so plugin authors can react but it should be removed now that Dakar is in beta.

Writing the message ONCE and for all on the plugin development topic for each plugin in the list above is the right approach. Yes. I can help with that!

-- TWiki:Main.KennethLavrsen

Some more ideas:

  • printing the message (to the log and to the screen) when configure discovers the plugin.
  • Have an "offline" checker that can be run by developers

-- TWiki:Main.RafaelAlvarez

Crawford: I think the new spec does not address the needs. It is not KISS and hinders administrators in a production environment (as Kenneth describes). I really think that Dakar should not check and report Dakar deprecated functions at runtime, period. If, then just when configure is run, as Rafael suggests. If you want the attention of the Plugin developer you can do that prominently on TWiki.org in the Plugins' home and dev topics.

Anton: On backward compatibility vs innovation (or progress), I firmely believe that innovation can be done with compatibility in mind. I draw a line between between innovation and desired architecture. That is, developers naturally think that backward compatibility hinders innovation, where in fact, they confuse innovation with nice architecture. Backward compatibility goes at the cost of "my preferrred architecture", but not necessarily at the cost of innovation. I certainly would like to see our developers focus more on user centric innovation. For example, the time Crawford just spent on changing the warnings (away from KISS) would be better spent on important innovations, such as a generic web services interface, a powerful rest interface, a J2EE/TWiki integration, syncronous edits, OO content access syntax, web based updates of TWiki/Plugins/apps, relational database backend, portlet integration, etc. All this can be done with a platform that is not ideal but "good enough". All this will draw new users to the TWiki platform.

Lastly, if we create a nice architecure at the cost of compatibility we take away the barrier to switch to a competing product. If the cost of upgrading TWiki is the same as switching to another wiki we will inevitably lose users.

-- PTh

The mechanism I implemented is required so that Dakar-compatible plugins can skip registration of deprecated handlers. So the only debate here is regarding the warning messages.

Checking deprecated handlers and Func calls when configure is run is not feasible. Plugin code is not executed in configure, and even if it were the infrastructure to perform the check would be a PITA.

At some point very soon we will have to remove out the deprecated handlers, because they throttle the rendering loop. You may be happy for large numbers of plugins to go down in flames unexpectedly at that time, but I'm not. We have to give plenty of highly visible warning. of course, you might be able code up an extra-chunky Func layer to compensate, but the performance of that layer is likely to be awful.

Clearly my approach is unpopular. On past performances, messages on twiki.org, and even direct mailings are a waste of time, so I await with interest the realistic alternatives.

frog.jpeg


OK, total lack of feedback suggests I won this argument, so I'm closing it.

CC


Lack of feedback? We all wanted you to remove the warning and we were waiting for you to do it. You cannot say you won the argument by putting a picture of a frog.

Please stop flooding our logs no matter what reason.

Re-opened!

The fix is easy - remove this line from plugin.pm

                $this->{session}->writeWarning(
                    $this->{name}.' defines deprecated '.$h );

But I don't want to start hacking my Dakar - because I will have to re-hack it each time an update is made which overwrites my modified files.

Flooding logs is - and has always been - a security issue because you drown the real danger signs from potential attackers in MBs of cruft.

And as I already said - people on a hosted system - will not appreciate to use up their 50 or 100 MB space with useless logfiles.

Those two reasons should be enough reason to remove the warning. Even without doing something else instead.

-- TWiki:Main.KennethLavrsen

I guess I didn't make myself clear. The warnings are there to act as a "big brother" for plugins authors, to apply pressure from the user community to get them to fix plugins. I said above that I was open to any other realistic alternative approaches to getting plugins fixed; I still am.

Removing the warning is not a realistic alternative. It is ignoring the problem and hoping it goes away. It effectively pushes the responsibility for plugin maintenance back on the TWiki developers.

Closed again, until someone comes up with a realistic alternative approach to getting plugins fixed as the API inevitably changes.

CC


May I step in to support CC's argument. As I am an contributor that scored rather bad in the topic here still not providing the fix to get away with the warnings wink I must say that those plugins that I fixed so far are better now.

Technically speaking, the plugin code is cleaner and more efficient as the core api is more efficient now. In fact the warnings and the extra gear that has been introduced by CC to switch off calls to deprecated handlers is a real speed improvement. There's been no commits to the core that changed anything else. The message still is the same: either fix the plugin or get the warning. Message received.

The policy you like to establish to get plugin authors (like me) to do my work is up to you. Flood me with emails, file bug reports, send me patches or even money. I will appreciate any feedback. So will any plugin author still interested in his/her work. If you are using an abandoned plugin whose author is not responsive at all, well, take over maintenance. Do we have to talk about the FOSS way of work any further. Think so not.

Baseline: read the code, that has been committed due to this bug report. It is good code; it helps to get things straight. Dakar is getting better and better!

MD


That is not the point MD. The point is that you screw the users and the system admins by flooding our logs to send a message to a few plugin authors.

This is not a discussion pro or con making API changes. I am not arguing against API changes. I am arguing against log flooding.

It is addressing the issue that in ONE MONTH by logs will be flooded by a 2 maybe even 3 digit MEGABYTE warnings that I can do nothing about.

It is addressing the issue that this flooding is a SECURITY ISSUE because you defacto hide any signs of attackers.

I can understand that you have the warning while Dakar is still not released. But this bug report should then be kept open until Dakar is formally released and the warning code is removed. I will for sure remove the code line from my installation.

It really worries me that you cannot understand the security issue related to log flooding.

-- TWiki:Main.KennethLavrsen


Both sides are right. I don't want flooded logs. The need to flag the deprecated stuff is important. How to do both at the same time, since they're both clearly needed? A third option must be found. Suggestions...

  • Opt out In the install stage, emphasise to the installing person that this stuff gets logged, and it may take up a fair bit of space. Offer them the option to turn this off in configure. This way it's a conscious choice. Pro: it's left on and developers will probably (hopefully) choose to make use of it. Con: many (most? all?) will probably just turn it off. It's possible that both will happen, which is what you want.
  • Reduce Keep logging the messages, but find a way to reduce the number. We want the messages, we just don't want the flooding. Only log them once per (hour|day|whatever). If this is possible, offer the chance to set the frequency (but not turn it off) in configure. Or maybe only print one log line for multiple messages, but this may be more of a PITA for various reasons. Pro: you log the warnings, just not too many. Con: ?
  • Redirect Keep logging them, but log them to a separate file. Doesn't solve the problem directly, but people can then deal with the log file differently: delete it periodically (without trashing the rest of the logged info) or whatever. Again, maybe offer some sort of option to deal with this consciously in configure. Pro: highlights the deprecated issue further, offers a (probably clunky) way out of one big logfile. Con: another logfile to deal with, but in this option that's the pro...
If this is an important issue, a third option will turn up.

-- TWiki:Main.MarcusLeonard - 05 Dec 2005

If you want to get the attention of the plugin authors, put a notice on the one page they care about: the plugin's home page in the Plugins web.

All information relating to whether someone should want to download the plugin should be there: appraisals, editor's picks, number of times the plugin has been downloaded and API cleanliness.

-- MC - 05 Dec 2005

My warning log continues to be more and more flooded. It is unacceptable. When I scrolled through the log this morning I saw several warnings drowned in the endless repetition of a warning I cannot do anything about.

Please try and see this from the view of the twiki user instead of the developers.

If you want to make a visible warning add the warning to the %FAILEDPLUGINS feature instead of flooding logs.

That should be fairly easy to implement and it also help me as an administrator to quickly assess if a plugin is worth putting in production or not.

The plugin that currently floods my logs is the Blacklist plugin. I was told that it would be EASY to fix. I have looked at it and I do not understand the very limited plugin API docs which I am not even sure are up to date. I do not find it easy to fix someone else's plugin being a person that has very limited knowledge of perl.

Professionel programmers often forget that they have a long education and years of experience and forget that a Twiki sys admin is most often not a programmer.

But I (a hardware engineer) found out how to remove the code line that produces the code in the first place. That was easy. Just search for "defines deprecated" in all files.

In lib/TWiki/Plugin.pm lines 216-217 remove the lines that says

                $this->{session}->writeWarning(
                    $this->{name}.' defines deprecated '.$h );

Is this elegant? No! But it removes the problem and allows me to see the important warnings that could very well be important to the security of my webserver.

I am sure many like me will do the same if this bug is allowed to pass through to the released version of Dakar.

Attackers trying to break through defenses can often be seen in the logs and I inspect my system, apache and twiki logs regularly. The flooding of logs makes it close to impossible to inspect the twiki logs because they are two digits of megabytes of useless repeating deprecated messages.

If you want a warning in a truely visible place put it in the %FAILEDPLUGINS feature where is does not harm anyone but becomes very visible. I am sure plugin authors always look at the output of %FAILEDPLUGINS.

-- KJL - 29 Dec 2005

I find the current solution also not acceptable. It is one item that was set to be reviewed in the TWiki:Codev/DakarReleaseMeeting2005x12x17. There are better ways to alert the Plugin authors without punishing the TWiki administrators.

-- PTh

Jolly good. As soon as you implement one of the "better ways to alert the Plugin authors", we can remove the warnings. But not before.

CC

It took me 2 hours to figure this out (I am only a Perl beginner). But I have attached a simple diff that solves this issue in an elegant, clear way and without jeopardizing security by flooding logs. It also removes the log flooding.

http://svn.twiki.org/pub/Bugs/Item1025/fix-1025.diff

I am sure Plugin authors will always look at the output of %FAILEDPLUGINS and they will see their plugin using a handle marked very clearly as deprecated.

And administrators will see it too so they can think twice before they put such a plugin into production.

KJL

Kenneth, that is an excellent idea - I couldn't see the wood for the trees, and missed your solution completely. I will put that into production immediately.

SVN 8060

CC

I am very glad that this contentious issue is resolved smile

-- PTh

Doc in EmptyPlugin.pm is out of date with this update, reads:

NOTE: Defining deprecated handlers will cause a warning to be output in the warning log file. See TWikiPlugins for information on guarding deprecated handlers that are defined for compatibility with older TWiki versions.

SVN 8167.

SP

ItemTemplate
Summary Flooded with warn messages from misc plugins (with fix diff attached)
ReportedBy KennethLavrsen
Codebase

SVN Range

AppliesTo Engine
Component

Priority Low
CurrentState Closed
WaitingFor

Checkins 7630 8060 8167
Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatdiff fix-1025.diff r1 manage 1.4 K 2005-12-30 - 23:41 KennethLavrsen Kenneth Lavrsen's safer and user/administrator friendly way of reporting deprecated plugin handlers
Edit | Attach | Watch | Print version | History: r40 < r39 < r38 < r37 < r36 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r40 - 2006-01-07 - SteffenPoulsen
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2018 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback