Wiki Development

This page discusses new technical features for the Wiki system the site runs on.

Pending Enhancements

See /Archive for existing enhancements.

To do:

Make generated pages validate as HTML 4.01. (Wookee's output seems to be okay, just a couple of links using unescaped "&" characters and some stuff around the ad <iframe> aren't.)
Fix the subpage-with-parentheses page title bug
Add a link to the login page at least at the bottom of every page (next to the Preferences link).
Links to pages that don't exist yet are not put in [brackets like this] but underlined (though not clickable).

Tarquin: You mean just the "?" is clickable? I presume you're just giving them a class & using the glocal stylesheet?

Mychaeel: Yes. I did that on JDN.
Get rid of the [brackets] around external links and replace them by a small prefixed icon instead. Does anybody have a concrete idea for that icon?

Tarquin: How about the little globe icon I put in the ScreenWebPage icon?

Mychaeel: Sounds good. I'll try that.

Tarquin: There's a really nice world graphic for Chimera: http://www.mozilla.org/projects/chimera/

Game Logo Smileys

Game logos to be included in a smiley-like manner are a great idea. Could also be used next to the usual UT or UT2003 on class pages, if only to spiff them up visually.

Basically all we need to add that feature are small (say, 16x16) GIF files with transparent background showing the respective game logos and named ut.gif, ut2003.gif, devastation.gif and so on. Then we'd only have to drop them into a special directory via FTP and we'd be good to go. Anybody up for the task of creating those icons?

Icons made so far (16x16, Alpha-channel PNG)

UT2003 icon:
UT icon:
... other game icons to be made by someone else!

NOTE: IE for Windows does not support PNG alpha channels correctly. So you'll get "ugly but functional" until you change to another browser. (IE for mac (5.0+), however, does support alpha channels. So does Safari for all you mac users out there and of course also mozilla, netscape and everything else gecko based. See [PNG-supporting Browsers] for an (hopefully) up-to-date list of which browsers support png)

Syntax

I'm thinking we need two types of syntax:

for class headers, simplest thing is to parse the "UT2003 ::", and prepend the logo.
elsewhere in text, something like ::UT2003::

Mychaeel: Hrm. Please explain why it has to be something as strange as "colon, colon, name, colon, colon." I'm already not exactly thrilled of that pseudo-C++ "name, colon, colon" that is being used in class path headers, but I can live with it. – If it's just for inserting the game logos, why not simply drop them in the "emoticon" directory and reference them as :ut2003: or :devastation: as originally suggested above? It's not even required to change anything about the script for that.

Tarquin: Fair enough it was just a rough suggestion. What would you prefer for class headers, BTW?

InterWiki

Tarquin: I plan to get round to making an icon to mark off-site links. Can anyone come up with a BuF icon for forum links?

CH3Z: There is the new posts symbol from BuF ...

Tarquin: The Vb symbol is good I think I'll upload it later

Timestamp Placeholder

CH3Z: here's another idea for the Wiki code... a timestamp function, unless there already is one and i missed it. nite

EntropicLqd: Timstamp function? Like the one displayed on the recent changes page?

Mychaeel: I think he means along the lines of the convenient ~~~ shortcut – just a placeholder that is replaced by the current date and time upon saving a page. Could be done; I'll try to keep it in mind until next time I'm changing the script.

Tarquin: let's just copy Wikipedia syntax: 3 ~ is name; 4 ~ is name + date

Mychaeel: Where a timestamp is needed frequently a name tag won't (like on a [Developer Journals]? page). I'd rather introduce, for example, %%% for a timestamp (anywhere in a paragraph).

Tarquin: Fair enough

CH3Z: hehe i'm such a nube. I never causght that 3 ~ bit since it changes the code on exit. Can you imagine how hard i would have been working if it let me put in the name i tried to use when i registered: ][C1][C|H|3|Z all that with 3 ' before and and after and the colon.

ZxAnPhOrIaN: That sounds good. I think that ``` would be better because it uses the same key to to ~~~ but you don't use shift. it would be much easier to access.

Tarquin: On YOUR keyboard, maybe

ZxAnPhOrIaN: maybe not!

Wormbo: Actually ` has weird effects on German keyboards: You have to press another key to make it appear and you still have to use the shift key. ~ happens to be on the same key as +, but us Germans have to press AltGr (right alt key) to use it. Choosing characters for these things isn't as easy as you might think.

ZxAnPhOrIaN: You live in germany wormbo too (I know Mych lives there)?

Pop-Up Search Window

Mychaeel: Something I'd find very handy, especially regarding the Wiki-like lack of a tight convention for naming pages, would be another link on the page editing pages that opens a small pop-up window with a page search field (and potentially the option to either search page titles or do a full-text search). That'd make it so much easier to create proper links to existing pages...

Tarquin: I'm starting to think a search on just page names would be good as a pop-up. Full searches are taking longer as the site grows.

Anchor Links

(moved discussion... is it worth keeping?)

Mychaeel: I've been wondering whether there could be a Wiki-like way to link to a particular header on a page. They're already automatically given anchors (for the QuickNav menu), but those anchors change when new headers on the same or any higher level are inserted before a header. Any thoughts?

Tarquin: You mean something like [[MyPage#2.1]] would jump to that anchor? Update: there's a patch for this on UseModWiki. Since our pages already have the anchors built-in, this should be simple. I'll look into it once the CVS is up

Githianki: Maybe I suck at properly organizing and thus desire these even though everyone seems to have gotten by without them, but they seem like they'd be really useful.

Wormbo: The problem with Wiki pages is, that they might be changed radically. Anchor links wouldn't point to the correct location anymore if someone inserts a new heading at the top of the page.

Mychaeel: ...and if you need anchor links, you'd better start refactoring a page into multiple smaller, more specialized ones (or a page with multiple subpages) to start with. :-)

Githianki: so maybe this is way off since I don't know how anchors are implemented at the coding end but if you eliminate resequencing anchor refs (which seem to be numbers based on what I am hearing), then the links will be valid unless you change the headings or delete them? I mean if you tie a value to a heading then don't change the value. You'd need to change the nav bar to sequence anchors in a non numerical order, perhaps by doc position. I don't know if the arnchors themselves are resequenced each edit to go 0..n or whether new anchors just get the next higher value than the last anchor/header placed. Um. This sounds stupid doesn't it? Basically I am saying, (if there is a facility for renaming links throughout a wiki, Whosit→Whatsit) when a heading is created assign it an anchor token that comprises both its text value for links and its position value on the page it appears for quicknav. The text is treated internally as a link (so it shouldn't matter where it appears in a doc), and its position value can be updated if it is moved around since that is only a local ref for quicknav. When a subheader is edited the text value (being treated as a link) would propagate its changes to all docs linked to it. This requires a type of doc path and costs overhead but do these things change so much all at once as to be a burden? Does any of this sound like more than drivel? Obviously this requires code support and depends on the presence or ability to make present a link renamer function (essentially a page renamer that updates all references in the wiki). Kill me if I deserve it

Mychaeel: The whole HTML code is generated from the Wiki markup source each time a page is viewed. At that time the header anchors are generated, and so is the "Quick Navigation" menu where they're (currently) exclusively used. – We could introduce a special "anchor" markup tag, of course, or automatically generate anchors from the text of headings (something like #anchor_links for the previous heading, for instance).

Topic hierarchy

Do we need to give "topic hierarchies" – see Event for an example. Would it help people find their way around?

And do we need markup to make a different sort of CSS box to @@ ?

Problems

Edit/Renaming Collision

Mychaeel: I just noticed: While I was renaming some pages somebody had another page with links to those renamed pages open for editing. When he/she saved that page, the automatic link adjustment done by renaming the pages was undone on that page. – Instead, an "editing collision" error should occur, I believe.

New Templates

EntropicLqd: Is it just me or has the "cool blue" style sheet suddenly broken on IE? It feels like the style sheet is no longer available on the server.

Tarquin: we're down to 2 content stylesheets at the moment – see project skins

Legal: Using IE 5.5, b0rked completely, yes...

Entropicqd: IE 5 seems OK although weird things happen when you edit text within the "text area" - it grows beyond the edge of the screen for some reason. IE 6 seems to be completely lost. I've taken to using the Phoenix browser for looking at the Wiki now to avoid these problems. It does the job but it's too slow to use all the time.

Tarq: I see what you mean in IE 6. The textbox grows by a few pixels each time you type, and then goes back to normal on a window resize. Not sure what to do about that, but I'll look into it. I don't have IE 5 – anyone know where I can download it?

ZxAnPhOrIaN: Ent, I have that same problem, i have IE6.0.2800.1106, to be exact!

Tarquin: FIX: turn off "Use 100% wide edit area (if supported)" in your prefs. I'll look into a better fix later.

CH3Z:THAT, ZxAnPhOrIaN, is what i was trying to relate to you at the sandbox. I saw the option when i registered my name, but don't see it anymore.(ok found it, "Selective Blindness" ty tarquin)

Advanced Search

Pingz: I don't think i've yet to look for information in the wiki by browsing categories. I pretty much use search exclusively. Try doing a search for the Trace function and see how many non-related results you get. Maybe some advanced search features can be added. Whole word only searches ( searching for Foo will not return FooBar ) and Boolean searches ( Foo AND Bar ) would do wonders. Maybe even add the matching word or phrase next to each link in the results list.

Mychaeel: The search returns those pages where most of the search terms matched first; so that effectively means it tries "and" first, and gradually falls back to "or" if it doesn't find anything.

Pingz: I guess then knowing if a result is an "and" or an "or" would be nice. Alot of my searches require "and" and the "or" results are just noise alot of the times. I enter multiple words into the search because i want a result with all those words. As it is now i then have to click thru links do a page find to see if the words i searched for are in the page at all... it's a frustrating process for finding information. Searching should be a very powerful tool within the wiki especially since things aren't always organized correctly. IMO the searching is not at the moment.

For example. I did a search just now with the words map, triangle, and count. The first result is Adding Polish To Maps. This page contains the word map and count, but not triangle. It was a waste of my time to check it and i would have been better off if i didn't get back any results.

I'm not trying to be a pain or nothing, but just trying to make the wiki a better tool for everyone.

Mychaeel: The search returns only the hits with the most matching words, first for page titles, then for the page content. So if the first returned page contained only two of your three search terms, that's as good as it gets in the following pages either.

If I get around to it, I'll think about improving the search function (which has already been vastly improved over the standard one in the unpatched UseMod Wiki script which doesn't even recognize multiple search terms). I don't think that'll happen anytime soon though.

Pingz: Here is an idea. What about using Google? Taking the same example from above and entering it into Google gives me exactly the results i want. Click [here]. You can easily replace the current search box with one that uses Google with the instructions [here]. You can also customize the results page some ( i think just coloring and adding the wiki logo ).

The advantages of using Google is great: lots of search options and highlighted results. The big downside is that the Google index is only updated about once a month... so results could be pretty stale.

Mychaeel: ...and outdated search results. And it's an external site.

Wormbo: A little suggestion for the search: Try searching for finding the local PlayerPawn, you'll be suprised at the uselessness of the search results although the correct page is somewhere there, too. The result for "finding the local PlayerPawn" is only the desired page, though.
As a result of this I have two suggestions: If the words appear in the exact order they were types in, a search result should get a higher priority and if the words appear in a heading this should probably also boost the priority of the result a bit.

Mychaeel: ...and if you remove the "the" from your query you'll notice how you just get a small handful of pages, with Netcode Idioms being in plain view. (The results listed before it are those where any of the search terms appear in the page title; so there.)

The search function would need some lexical analysis of the entire Wiki to be able to automatically give the "the" a low priority so that entering "finding the local playerpawn" doesn't return loads of pages with "the" in their title. I doubt that can be done with reasonable effort.

Mychaeel: Well... obviously the Wiki search function is a vital part of the site, and even though the way it's implemented right now represents a huge improvement over the original UseMod Wiki search functionality, it's still subpar for a site like this. The biggest problem lies in the ranking of the search results. Performance is not yet an issue, but might become one as the site grows and the search function becomes more complex.

I have some ideas how to algorithmically approach that problem, but to my chagrin and mild frustration I'm not going to have time to really wrap my mind around this during the next weeks until early October. Nevertheless, here are some rough notes (subject to change):

Search function provides a means to limit the search to certain categories so that people can directly specify that they only want to get results for UT2003, for example. Naturally the list of category tags has to be cached somewhere so it doesn't have to be built from scratch each time the Search page is displayed.
Automatic compensation for differences in American/British English spelling: The search engine is provided with a list of American/British English word pairs and internally "normalizes" both the search terms and the searched pages; so a search for "colour" would also find pages containing only "color."
Page ranking is based on...
- Relative position of the individual search terms to each other. The page is scanned matching term by term, and for each term a ranking bonus is added depending on
  - how many terms in the search term list have to be skipped from the last one to get to the next one (ideally: none), and
  - how close the terms are to each other on the page (ideally: directly following each other).
- Absolute position of the search terms on the page (still in relation to the page length though). Each matching term adds a bonus to the ranking depending on its position in the page. The page title gets an extra bonus here.
- Distribution of the search terms on the page. For that, the algorithm could try to split the page in a row of "sections" of equal length and try to find the smallest section length where still every section contains at least one search term. The more sections are possible (that is, the smaller a single section may be), the higher the ranking bonus.
- Perhaps logical markup: Terms in headings count more (depending on the header depth). That'd involve some sort of markup parser in the search engine though which I'm not too fond about; but if the search engine is optimized by caching a keyword list for every page that's automatically updated when the page is saved, the page content could be simply piped through Wookee before saving it and Wookee could provide the parsed structural information that's required for this function.

inio: Before anyone suggests it, back-reference-based ranking is off-limits, unless someone wants to work with Google to get a license for the PageRank (orig. Backrub) patent (which pretty much covers using back-references to rank results of a search in a document-reference network).

Non-Existant Pages

Mychaeel: I'm pondering something I've seen on [Everything2]: A link to a non-existant page doesn't directly open its edit page but performs a search on the page name's space-separated parts first and shows those results. It also gives the option to click a special link to actually create that page.

That patch would basically involve creating a magic page that explains all that, lists the search results and gives the actual edit link; and changing the script function that returns the "?" edit links to return a link to that magic page instead, somehow passing the desired page name as a parameter.

Tarquin: just seen in the script there's a $NotFoundPg – you can make requests for pages that don't exist go one particular page. As for the edit link to a non-existing page, having that go to plain text first is done on some wikis. Eg Dive in Os X. We could do it here, I suppose it depends on the balance between preventing n00b experiments and irritating power users

Contributor News

Kerlin: I hope this is the appropriate page for this discussion but how about a secondary news page of wiki contributor news? Zxanphorian could post when a new map is available. Mychaeel will post to it when Jailbreak is compelted (though we'll see that on all the news sites). El Muerte TDS would post when a new rev of UnCodeX is released, etc. Otherwise, you have to fish around a little bit to see what each individual contributor has done recently. I think the rule would be "don't post to the contributor news page unless it's a public release". Like any news site, descending order by date. Every two weeks you get rid of the old stuff.

Mychaeel: Actually that's what the Featured Pages page is for that's linked from the Home Page. Just nobody really uses it yet.

Kerlin: Ahhh. It looked like Featured Pages was probably what I was looking for but, as you said, since no one was using it I figured I was mistaken.

Site-Wide Bans

Mychaeel: Changed the banning mechanism of the Unreal Wiki after I found to my surprise that it only forbade edits, not access. Now it does.

E-Mail Notification

MythOpus: Hey Mych, how does e-mail notification for the wiki sound to you?

Mychaeel: Like a lot of work that I really don't have the time for at the moment. ;-)

The idea has potential, but it's also very open for abuse. The Wiki would need a way to validate the entered email address (given that we'd add such a field to the "Preferences" page to start with), and while that's definitely doable following a similar scheme as vBulletin, is entails quite a bit more development than just adding a small feature.

Tarquin: It's already a feature of UseMod logo UseModWiki. It needs a mail server – which AFAIK we don't have available.

Mychaeel: We definitely could send email from here. However, I don't see that this feature includes any sort of email validation, so it's far too open to abuse to be a viable option.

Offline Readers/Bot Limiting

MythOpus: I'm not an expert at all this bot and offline reader stuff, but wouldn't it be possible to limit how many offline readers/bots are allowed on the server at any given time. Maybe if you install limitors then maybe you can 'un-ban' these things?

Tarquin: I seem to remember we thought the simplest thing would be a concealed link to a "I'm a robot, ban me!" page.

Mychaeel: In case that wasn't clear, one single offline reader firing requests as fast as it can is able to grind the servers to a halt. The incident a few days ago was such a case. It didn't even stop after it had been banned and only received "Forbidden" messages anymore.

That aside, you can't distinguish an offline reader from a legitimate browser – they both just request pages. And then, even if we could, any mechanism we could implement would have to be in the Perl script itself – and when an offline reader hammers the server with requests, it doesn't make much of a difference if the script then delivers a "You are banned" message or the actual page.

And I don't see the point in using an offline reader to harvest the Unreal Wiki. We're providing Offline Wiki after all.

MythOpus: OH I See.... That makes sese and I was wondering about that offline reader thing. How amny times does the offline Wiki get archives?

Mychaeel: That's all explained on Offline Wiki.

Technical vs. Specific

Foxpaw: I've noticed lately a lot of pages being made that document people's own creations or other third party classes/content. Should we tag or otherwise identify these types of pages so that people will be aware that they are not discussing something in the base UT/UT2003 distribution? I think that it could lead to some confusion if people end up on a wild goose chase trying to find out what happened to a class that they believe was included in their base classes but was actually a custom class written by someone else.

Tarquin: tag with "Category Custom Class" and make it clear in the opening paragraph that it's a custom thing. You might also specify the package in the header box.

Banner Ads

GRAF1K: Has anyone else noticed that, in Mozilla 1.4.1 at least, when you scroll with the mouse wheel up and down around the banner ad it slides down off the page?

Gehn: I have noticed that exact thing. In fact after reading this posted I wanted to tell them about it and I had to reload the page because it did that slide thing.

Hierarchal markup

Sobiwan: Is there are markup that displays hierarchy lines such as found in the Class browser? There are many pages here that use ASCII markup for this purpose.

Tarquin: Interesting idea... but I think there would be too many problems to solve, such as making it work at all font sizes.

Sobiwan: I think it would work... unless I am missing something. It would be similar to bulleted hierarchy because the markup is inherently indenteded, so it would keep the formatting at any font size. Instead of using bullets or numbers, it uses lines and plus signs like the ASCII counterpart.

Tarquin: What sort of markup do you envisage? What sort of output?

Sobiwan:Sobiwan: Since markup is supposed to affect layout and people are using ASCII to create hierarchal layout such as found on the Actor Class Hierarchy page, it would be good to include this type of layout as a markup to keep the concept of markup lauyout consistent. One could create a markup such as:

<hierarchy>
 Actor
   +- AimedAttachment
   +- Brush
   |+- Volume
   +- Controller
</hierarchy>

The WIKI would display the lines nicely, auto indent sub-items, auto indent word wrapped sub-items. It doesnt have to be collapsable, just make good looking layout instead of relying on ASCII.

Tarquin: well what I was thinking you'd want to write was this:

+ Parent
++ Child
++ Child2

It could output the sort of ASCII we currently do. I don't know how we can "display the lines nicely" in HTML.

Sobiwan: The only requirement for markup is ease. Perhaps the markup would generate a dynamic table or CSS formatting to get lines, colors, indentation and word wrap.

Tarquin: I think the current ASCII tree is fine for readers. The problem as I see it is that it is a pain for writers to create. It might be possible to create a tree-type thing with CSS but it would be a huge mess, unless there's something I've not thought of. This is all academic, as it's Mych's Wookee that has to be altered and I think it's beyond my perl skills.

Wormbo: Let's look at a typical ASCII art hierarchy:

 Parent
  +- Child 1
  |   +- Child of Child 1
  |       +- Grandchild of Child 1
  +- Child 2
      +- Child 1 of Child 2
      |   +- Grandchild of Child 2
      +- Child 2 of Child 2

That could be created with Wiki Markup like this:

+Parent
++Child 1
+++Child of Child 1
++++Grandchild of Child 1
++Child 2
+++Child 1 of Child 2
++++Grandchild of Child 2
+++Child 2 of Child 2

Wookee only has to watch out whether a vertical line is neccessary or not.

Tarquin: That's what I envisioned too. But we seem to have lost Mych – I emailed him last week to see if he had any ideas about our google problem, and no reply.

Mychaeel: Heh – I replied to your email yesterday. Please check your mail before complaining. :-)

Tarquin: Hi Mych! The system would be quite complex – a certain number of "+" need indentation with spaces, plus possibly vertical lines, depending on whether there are further siblings (that is, a line with the same number of "+" before the next line with fewer "+" or the end of the block. There's also the matter of the whole block needing PRE tags, but not the sub blocks. All in all, it needs the Wookee-master. But I would say this is lower priority than Jailbreak

Sobiwan: Yes, there is no rush on this. I wasnt arguing the ASCII tree is good or bad for readers or writers, only that I noticed it was not following the rule of markup for layout. I like how the markup idea has been refined; its easy to follow.

CSS being customizable, there should be a way to define each level of a list to display the vertical and horizontal lines. Although I'm rusty at CSS, I can take a stab at getting the layout right using simple CSS code, then Mych can figure out how to translate it to Wookie.