UESPWiki:Community Portal/Semantic Wiki

This is an archive of past UESPWiki:Community Portal discussions. Do not edit the contents of this page, except for maintenance such as updating links.

This page has been given archive name but not status as the subject is suspended until The Elder Scrolls Online has been dealt with. You are free to continue the discussion and ignore the notice.

Suggestion: install the Semantic Mediawiki Extension

I believe it would be a good idea to add the Semantic Mediawiki Extension to the wiki and use it in the upcoming ESO content. It provides the functionality of "saving" attributes of items much like our MetaTemplate extension does, but it does so in a more transparent manner (there are all sorts of reporting tools, including an optional infobox at the bottom of pages to show to editors who have it enabled what values a specific page saves). Another big advantage is that other people [1] will be responsible for keeping it compatible with upcoming MediaWiki versions.

Since most of the MetaTemplate use is done via templates, the scope of an eventual project to convert the uses of Metatemplate #load and #save to SMW seems manageable. Another gaming wiki which has integrated SMW successfully is the official GuildWars 2 wiki. --◄mendel► (talk) 04:25, 12 February 2014 (GMT)

I've come across Semantic before and at the time, I didn't think it was quite ready to take over for what we were using MetaTemplate for. That was some time ago, though, and I'd like to have another look at it. You have a point that ESO might make an ideal conversion ground, since so much of it seems likely to be different that new templates may be needed for a lot of things anyway. – Robin Hood  ^(talk) 06:34, 12 February 2014 (GMT)

Could you give some examples for features that we'd need that you didn't find in SMW? --◄mendel► (talk) 12:26, 12 February 2014 (GMT)

Switching would be a major change, it will not be as simple as spotting single features missing. Currently we rely really a lot on MetaTemplate, so minor differences in behavior will affect us. But in principle switching might be a good idea. Currently we still have caching problems with MetaTemplate - after editing a page saved values might not show up on pages loading them. We may end up just trimming down MetaTemplate and still use some of its features (for example, what about #inherit, looking at variables defined further down the parse stack?). I think, if an initial look at it is promising, we could install it on dev once the test for the upgrade is over, to gain some experience with it. --Alfwyn (talk) 12:54, 12 February 2014 (GMT)

I'm working off these assumptions:

1. The Metatemplate functions #load and #save are used to store and load object attributes. (I assume this because I have read the extension docs and seen some templates using this.)
2. SMW can replace this functionality. (I assume this because SMW was designed to do just that, and because it works for GW2W.)
3. If SMW can replace MetaTemplate, this might be beneficial (for the reasons stated).

You and RH70 have expressed some support for assumption 3. Before we can evaluate this, however, we must first examine the other two assumptions in detail. You both seem to tentatively challenge them, but you don't go beyond voicing general doubts. I have these doubts myself, because I don't know the wiki very well, and I'm self-critical enough to admit that I may be completely off-base with these assumptions.

I want to know where your doubts are coming from. I am asking for examples to figure out where you're coming from, and to go there myself. I want to understand what you're telling me, and I want to become able to assess my assumptions myself. I do work toward this goal on my own as well, but it would really help if you just mentioned some issues that you'd want to investigate. Add some specifics (not expected to be complete or final) to your general doubts, please.

And yes, this is about the Data Sharing features only (for now?). --◄mendel► (talk) 14:27, 12 February 2014 (GMT)

One example would be to conditionally load data {{#load:....|if=...}}. Details like this will need looking at, and later testing. We probably don't need a perfect feature match, as we can just change how we use them a bit. But the general doubts from me are just, that I think that switching is a bigger undertaking than it may look. I think SMW will be worthwhile, if we eventually can replace all data sharing of MetaTemplate and retire those bits of code.

And there will be a lot of decisions to be made. For example we could change MetaTemplate to internally use SMW. That may or may not be better than changing all our templates. --Alfwyn (talk) 15:08, 12 February 2014 (GMT)

(←) As an average site-user, I have no idea what any of this tech-talk means. However, as a member of the community, if this change will ultimately be beneficial for the site (especially with ESO coming out), then I think we should seriously consider this change. However, if there are things that will need to be sorted out and tweaked, I would say we should do this fairly soon in order to be prepared for the ESO release. •WoahBro^►talk 15:17, 12 February 2014 (GMT)

I don't remember specific features that were missing or problematic at this point, as it was at least a year ago, maybe closer to two, when I looked at it. It works on a slightly different paradigm by default, but the #set and #ask/#show would seem to be similar enough at a glance that I would expect adapting existing code to be straight-forward in many cases.

In addition to Alfwyn's point above about the if= (which is evaluated before any other parsing takes place), a couple of things that spring to mind off-hand that I didn't see covered on the website are: how well it supports namespaces and whether you can use variables within names themselves (e.g., {{#show: {{{ns}}}:{{{faction}}} | ?Rank{{{rank}}} }}). You'll see this sort of usage in template calls like {{Faction|Mages Guild|rank=Evoker}}. – Robin Hood  ^(talk) 18:52, 12 February 2014 (GMT)

Rudimentary testing on their sandbox wiki showed that property names can be indeed constructed. But thinking more about it, if we go SMW, we probably want to keep MetaTemplate syntax at least for some time and just let it store/load its data into properties. That will ensure that things like if= will work. And we can switch stuff over at a convenient pace then. MetaTemplate would need to auto-create properties (probably constant prefix + saved name), if not already present, of type text. That will not be as useful as typed properties, but will work even if the same name is used in wildly different context on different pages. Of course if we switch over to typed properties, we can ask more useful questions then (like give me all NPCs level greater 20), but cases of n/a may be a problem. --Alfwyn (talk) 21:48, 12 February 2014 (GMT)

As far as I can tell, we don't really need the if= syntax any more, because the rules about expansion of conditionals have changed years ago.

Are you proposing to rewrite the Metatemplate extension to use SMW? I believe it is easier (and more transparent) to just rewrite the templates (27 use #save, 40 use #load ^*) for SMW, setting up a consistent naming scheme across the wiki in the process.

My next job is going to be documenting the uses of #save, which is useful even if we don't go SMW. --◄mendel► (talk) 04:58, 13 February 2014 (GMT)

Yes, if it can be done without too much trouble, I think keeping the MetaTemplate syntax will be worthwhile. It keeps templates more readable. We have quite a bit knowledge about it, both encoded in the templates and in the minds of long time editors. Just switching to something else will change problems we know about (like a minor cleantable issues) to problems we don't know about.

Now retiring the internal load/save code of MetaTemplate will be worth it in my opinion. We currently have caching problems with it. I tried a bit to get to the root of it about a year ago, but ultimately found out I would need to know way more about how exactly the job queue works (that is, what is done in which order), and how exactly all caching works. Another thing that might get better is an optimizing issue. It looks like if I currently edit a page that saves data, all pages loading it get re-done, even if the saved data didn't change at all. This seems to escalate a bit. If I edit the "Book Summary" template for example, the job queue peaks at 5k. This may be in part to recursively changing stuff (all book pages, lore article referencing them after that, game space articles transcluding the lore pages, ...). I haven't really tested it, but it might be that editing an innocent page suddenly results in half of the wiki pages re-rendering.

The "syntactic sugar" part of MetaTemplate on the other hand is way less complex and currently doesn't cause any real problems. --Alfwyn (talk) 10:47, 14 February 2014 (GMT)

(←) My various thoughts on it for what its worth.

• Minimalism -- I generally try to install only the minimum of what is actually necessary/wanted. In general it is easy to add something but difficult to remove it later on. This is especially important for such a large extension as SMW...a lot of the smaller extensions can be easily added/removed with almost no effect to the wiki.
• Comparable Usage -- Looking at the sites that use SMW there doesn't seem to be anything close to the traffic the UESP gets (the closest was probably 1/10th of us at the moment). With ESO coming out and our traffic likely to see a peak x10 growth I'd be concerned about how SMW performs under such loads and usage. I would really prefer to see a site similar to us use it successfully and not be the proverbial guinea pig.
• Performance -- Trying to get the UESP as fast as possible has been a large part of my work over the years and I'm proud to see that we're usually faster than most wikis and other sites out there (currently around the 80-90% percentile). Looking at some of the other sites that use SMW I'm not impressed by their performance...all were slower and most were much slower, some as much as x27 (2.7s render time compared to our typical 0.1s). I might be willing to part with a small performance loss but not much if I can help it.
• Effort Involved -- For such a large and significant extension there would have to be a lot of testing work to see if it is compatible with the UESP. Performance would have to be tested exhaustively as well as its effect on caching performance and server load. Then there's all the work of actually modifying 20,000 wiki pages to take advantage of SMW. There's also the current backlog of important things which, given the release of ESO shortly, means I wouldn't have time to even begin looking at SMW before the fall (granted, some of the other admins could take a lot of the work).
• Actual Need -- Given we already have a extension (MetaTemplate) that has been created specifically for the needs of UESP I'm not sure I see any need to move to SMW, especially considering all the above challenges.

Now, the above list may seem entirely pessimistic but I'm always open things. If the other admins/editors see a benefit and can convince me I have no problem with it. Personally I think the effort would be better spent taking some time to look at and improve MetaTemplate from some of the issues its shown. -- Daveh ^(talk) 17:48, 17 February 2014 (GMT)

Thank you for taking the time to write such an in-depth reply. I can definitely see where you're coming from.

I agree that SMW is not the kind of expansion that you'd just install "to see if it works out". However, it works well for The Official Guild Wars 2 Wiki, which is has content that is very similar to ours (lots of pages about items and NPCs with stats, and lots of lists and other references to them), and GuildWars being a popular MMORPG, should have a lot of traffic. It is kind of hard to tell using MediaWiki stats, because page accesses often get to the cache only and not the server itself, so the statistics Mediawiki itself keeps are unreliable. I have looked for GW2W traffic stats, but couldn't find any. Anyway, page generation depends on edits, and looking at their RC, it seems as busy as ours.

Going by the last HTML line of each page, "Served in 0.110 secs" is a stat that GW2W can't even approach, being closer to 0.5s most of the time. It'd be interesting to find out how much SMW contributes to that. The actual time for a browser to request a page is actually slightly better, from where I am, but that's cache performance and not backend.

Effort - yes, it is, and since much of it rests on your shoulders, the decision must be yours. However, because SMW is not a homegrown extension, I'd expect few problems. To convert existing content, we ought to get by with updating the 60 Templates or so that are using #load or #save, and I volunteer to help with that. I was hoping SMW could already be used for the new ESO content, but given the testing requirements, I realize that it may be too late for that.

Ultimately, is it more effort to track down and fix the Metatemplate issues, or to go with SMW (assuming it doesn't have major issues)? --◄mendel► (talk) 15:41, 23 February 2014 (GMT)

Thanks for point out the GW2 wiki. They look roughly the same size as the UESP in terms of traffic and site size. For now I have other priorities to work on but when I eventually get to MetaTemplate/SMW I may try and contact the GW2 admins/devs and get their opinion on SMW. The performance difference may not indeed be due to SMW as it is impossible to tell by just looking at page loads externally. -- Daveh ^(talk) 16:48, 23 February 2014 (GMT)

Updating "60 Templates or so" is a not a thing easily done, unless we risk a more or less broken site. Those are all the central templates, small changes in their specification will cause unexpected results. And I see no point in obfuscating our templates, just so we don't use MetaTemplate anymore. Sure, we could just replace each "{{#load:page|data1|data2|...}}" by "{{#define:data1|{{#show:page|?data1#}}}} {{#define:data2|{{#show:page|?data2#}}}} ...", but that's not going to increase the readability.

Most questions about SMW would need actually testing, without that we can just take guesses. I wouldn't expect the render time to increase at all, since in effect both extensions will just add a few lookups in the DB to get the saved values. What might cause performance issues is querying complicated concepts, but #listsaved has basically the same performance risk issues. A thing that might actually change is the load induced by editing a page. MetaTemplate works by emulating transclusions, so changing a page #saving values will at least recursively invalidate the cached version of pages #loading those values (it looks like they are actually forced to re-render, I'm not sure why this is done). What would be useful to know is, what concept SMW uses to update and find pages needing update when a property changes - haven't found anything on that so far (but a page showing a property doesn't show up as transcluding the page the property is defined). If they do updates only if the value of a property changes (in contrast to every time something on the page is edited) it could actually increase overall performance.

Looking a bit at the SMW source, it is way more complex than MetaTemplate. Then again the hope is that most of the testing/fixing is done by other people (MW itself is quite a complex piece of code too).

My personal guess is, that it will be way easier to use the data storage of SMW in MetaTemplate than to fix its caching issues. I don't think we will have much initial use of inline defining of properties in links, but that may change once the possibility exists. Likewise I think #load/#save is superior to using #show/#set and just fundamentally changing the style of all our templates is infeasible in my opinion. We don't use #listsaved that much, and could probably just change over to SMW equivalents where we do. The ability to easily arrange saved data, and to easily find out where data is saved and loaded will be a bonus.

In summary I think SMW will be worth testing and several important questions about it can't be answered without. After that it will be easier to decide whether to use it or not. --Alfwyn (talk) 18:04, 23 February 2014 (GMT)

I agree that it would be worth testing at some point, but given how close we are to ESO's release, I think now is probably not the time. One thing I do remember noting on SMW's FAQ was this: Why doesn't data I have just added show up in queries? Sounds like they have similar caching issues to MetaTemplate. Perhaps it's a limitation of the MediaWiki software? – Robin Hood  ^(talk) 19:55, 23 February 2014 (GMT)

I guess I should define what I call the "MetaTemplate caching problem". It's actually not that it doesn't update the data fast enough. It's that when an old value already exists and I edit that value (or any other part of the page), pages loading that value get (randomly) broken by displaying no value at all. If it just would display an outdated value that wouldn't be nice, but somehow acceptable. --Alfwyn (talk) 20:03, 23 February 2014 (GMT)

Thanks for the clarification. I wasn't sure which caching problem you were referring to, specifically. That bug is something I'd like to take a crack at at some point, but I'm not sure I'll get any farther than you have.

Another potential concern of SMW is the database bloat. I just tried installing it on my test wiki, and SMW on its own created almost as many tables (38) as there are in the base MediaWiki software itself (50 for 1.21+). It's possible that they're all really necessary and useful, but that does seem a bit excessive to me, and even if it's keeping redundant data to a minimum, I have to think that that's going to be a much larger footprint (which often, but certainly not always, means less performance) than MetaTemplate's two tables.

Also, on the issue of examining the data, I'm thinking it wouldn't be too difficult to use the new (1.21+) Special:PagesWithProp as a model for a MetaTemplate equivalent. A companion Special page that would also be useful would be one that'll show all MetaTemplate properties currently defined for a page. Just a thought for possible future development. – Robin Hood  ^(talk) 23:07, 23 February 2014 (GMT)