App/Wiki types/Wikia.com

Acknowledgments

Many thanks to Nocte Draconis for helping to compile these instructions.

Overview

Wikia.com is a wiki farm with over 370 thousand wikis. Most of the material is made available under a copyleft license: Creative Commons Attribution-Share Alike License 3.0 (Unported) (CC-BY-SA).

As Wikia's wikis run on version 1.19.9 of MediaWiki, it is possible for XOWA to import their database dumps.

Note that the same process can also be applied to other MediaWiki wikis. For example, the Nethack wiki runs on MediaWiki and makes its dumps available for offline use:

• Article dump: http://nethackwiki.com/nethackwiki_current.xml.gz
• Image dump: http://nethackwiki.com/nethackwiki_current_images.tar.gz

The remainder of this walkthrough details the process to import a wiki. It assumes the following:

• A Windows machine with XOWA installed at C:\xowa
• A wikia of dungeons.wikia.com (Dungeons and Dragons)

Part 1: Set up the wikia dump

• Create a folder for the wikia dump. For example, C:\xowa\wiki\dungeons.wikia.com
• Download the Current pages dump from http://dungeons.wikia.com/wiki/Special:Statistics If a dump is not available, you can do the following
  • Log on to the site as a wikia user (You must create an account if you don't have one)
  • Visit http://dungeons.wikia.com/wiki/Special:Statistics You will see a button for Request a database dump
• Unzip the dump. You should end up with a file called C:\xowa\wiki\dungeons.wikia.com\dungeons_pages_current.xml

Part 2: Import the wikia dump

• Navigate to Dashboard/Import/Offline
• Change the Wiki choice from Wikipedia to Other wiki
• Change Where to get the dump from download to read from file
• In the next box, click on the ... icon. Select C:\xowa\wiki\dungeons.wikia.com\dungeons_pages_current.xml
• Click Import now
• Wait for the import to complete

Part 3: Import the images

This part is more complicated and requires three steps:

Part 3a: Setup dumpgenerator.py

• Setup Python 2: https://www.python.org/downloads/ . For the rest of this walkthrough, we'll assume it's installed at C:\Python2.7
  • Note that the dumpgenerator.py script will not work with Python 3
• Download the wikiteam project from https://github.com/WikiTeam/wikiteam For the rest of this walkthrough, we'll assume it's downloaded to C:\WikiTeam
• Setup kitchen and requests. Assuming Python
  • cd C:\Python2.7\Scripts
  • pip install kitchen
  • pip install requests
• (If Windows) Edit the dumpgenerator.py script

Old

        filename3 = u'%s/%s' % (imagepath, filename2)

New

        # replace invalid windows chars
        filename2 = filename2.replace(" ", "_")
        filename2 = filename2.replace("\"", "%22")
        filename2 = filename2.replace(":", "%3A")
        filename2 = filename2.replace("?", "%3F")

        filename3 = u'%s/%s' % (imagepath, filename2)

Part 3b: Run dumpgenerator.py

• Run C:\Python2.7\python C:\WikiTeam\dumpgenerator.py http://dungeons.wikia.com --images
  • Note: You can also download an up-to-date article dump using the "--xml" flag. EX: C:\Python2.7\python C:\WikiTeam\dumpgenerator.py http://dungeons.wikia.com --xml --images
• Move C:\Python2.7\dungeonswikiacom-20170201-wikidump\images to C:\xowa\wiki\dungeons.wikia.com\file\orig\images
  • (Optional) Delete any .desc files

Part 3c: Setup Imagemagick and Inkscape

• Get the portable versions of Imagemagick and Inkscape. See Dev/File/Setup/Windows, Dev/File/Setup/Mac OS X and Dev/File/Setup/Linux
  • For example, on Windows, make sure the files are copied to C:\xowa\bin\windows_64\imagemagick and C:\xowa\bin\windows_64\inkscape

Finally, run XOWA and navigate to dungeons.wikia.com/wiki/Drow_(4e_Monster). You will see images on the page

Other notes

• Broken wikis: Some wikis are missing pages. As of 2013-12-15, the Marvel and Yu-Gi-Oh dumps are know to be broken. You may have to contact Wikia.com to request that they investigate.

Note that broken wikis will show a lot of pages with "{{:Page_name}}". This indicates that "Page_name" is missing from the dump. For example, the Marvel dump will show {{:The_Wolverine}} because it is missing Template:The_Wolverine

• Missing extensions: Wikia.com uses many custom extensions that are different from Wikimedia's extensions. For example, the #var and #ask extensions are specific to wikia.com only. XOWA renders these as "{{:#extension}}" or "<extension>"

XOWA will try to implement these custom extensions in the future, but due to time constraints, they will be prioritized after Wikimedia extensions.

If you have an extension that you would like to see supported, please contact me through Help/Feedback and let me know the wiki and the extension. Depending on number of requests and difficulty of implementation, I will try to include support for it.

• Stylesheets: Wikia.com uses custom style sheets per wiki. By default, XOWA uses stylesheets for Wikimedia wikis. If you are adventurous, you can try to implement your own wikia.com stylesheet. Changes would need to be made in the following locations:
  • css files: C:\xowa\user\anonymous\wiki\dungeons.wikia.com\html\xowa_commons.css and xowa_wiki.css . See Dev/Design/CSS for more info.
  • html layout: C:\xowa\xowa.gfs. Refer to the lines after page_read =

You can refer to this thread for instructions on creating an inverted-colors css for Wikimedia wikis

If you create a css that matches the wikia.com one and would like it to be part of XOWA's release, please contact me via Help/Feedback

Developer notes

Resetting images

To clean up the images, do the following:

• Delete C:\xowa\wiki\dungeons.wikia.com\file\orig\^orig_regy.sqlite3
• Go to home/wiki/Special:XowaCfg?grp=xowa.files.general and press Clear cache