Blog/2016-04

From XOWA: the free, open-source, offline wiki application

Blog links

2019

2018

2017


Archives

Release: v3.4.4.1 (2016-04-24 19:30 Sun)

The desktop app is a trivial release. It has a few minor parser fixes, primarily for German Wikipedia.

The Android app is a minor release. It has one significant fix for not requiring one database: file-xowa. This will only be necessary for the German wikis

(Package) Update German wikis (2016-04): https://archive.org/details/Xowa_dewiki_latest

I updated the German wikis this week. You can find them at https://archive.org/details/Xowa_dewiki_latest Note the following:

(Android) Fix for missing file-user.xowa database

The Android app would crash if it was missing this database: *-file-user.xowa. See https://github.com/gnosygnu/xowa/issues/36#issuecomment-172292021 .

Here's the short version:

  • If you are planning to use the German wikis, or want to be prepared for future dumps (English Wiktionary), you should install v3.4.4.

Here's the long version:

  • XOWA desktop allows you to download the original versions of images for offline viewing
    • You can download these original versions by clicking on any image in an article
    • These images are stored in a file like en.wikipedia.org-file-user.xowa
  • XOWA's file database system requires that this database is always present.
    • If the file-user.xowa is not present, XOWA creates one automatically.
  • All XOWA packages on archive.org do not include file-user.xowa
    • I don't include this file, because I don't want to overwrite a pre-existing file-user.xowa when a user upgrades to a new package.
    • I was relying on the auto-create mechanism to create a file-user.xowa.
  • However, on Android, all apps have extremely limited write permissions. On an external SD card, they can only write to one directory: /<sd_card_mount>/Android/data/<package_name>
  • If there was no file-user.xowa, XOWA Android tried to create it. Because of the limited write permissions, it crashed when it couldn't do so.

v3.4.4 still tries to create the file-user.xowa file, but no longer crashes if it can't.

(Desktop) Minor parser changes and fixes, including using non-ASCII strings for gsub find

These are a bunch of minor parser fixes for German Wikipedia. They affect a small number of pages (<100) and don't affect content. The most interesting item is that it required another change to LuaJ. It mistakenly disregarded UTF8 byte encoding for bytes between 128 and 255. This caused gsub and similar calls to fail when using non-ASCII characters, like "ß"

Next release: v3.5.1

I'm working on a new wiki update tool which will make setting up the databases much easier on Android.

For wikis, I'm going to update the rest of the English wikis. I'm also going to try to fit French in.


Release: v3.4.3.1 (2016-04-17 21:00 Sun)

The desktop app is a minor release. It has various minor fixes, including broken Portal pages and a missing Wikis sidebar

The Android app is a major release. It has support for Kit-Kat. In addition, it has finally been released to the Play Store!

(Android) XOWA Android app on Play Store

The XOWA Android app has finally been released to the Play Store. Surprisingly, this involved a lot of unforeseen tasks, like graphic design, video recording, icon generation, secure signing, etc. Many thanks to anonymous4 for invaluable help, feedback and support.

Here are some useful links:

Finally, if you've been using any of the earlier Android App versions, I recommend you update to this version. It only has more UI tweaks in terms of the icons and Settings page, but it is the official release version.

(Android) Kit-Kat support for Android app

The other major item in the Android app release is support for Kit-Kat. This too was made possible by anonymous4 and provision of a Kit-Kat device.

If you have a Kit-Kat device, please give it a try.

(Desktop) Minor changes and fixes, including missing random sub-pages in Portal, missing Wikis sidebar in home wiki and no {{DisplayTitle}}

v3.4.2.1 has a number of minor fixes. In brief:

  • Broken sub-pages in [[Portal]]: Portal pages often display a Random sub-panel. These panels were broken due to an obscure XOWA Luaj bug (# of pages from int to long).
  • Missing Wikis sidebar in the home wiki: The Wikis sidebar did not show in the home wiki due to some CSS changes last week.
  • {{DisplayTitle}} was ignored: {{DisplayTitle}} was broken due to recent changes for PageBanner. This means that a page like http://en.wikipedia.org/wiki/gzip showed "Gzip" as the title instead of "gzip".
  • Script error for cycling tables on de.wikipedia.org: A few dozen cycling-related pages on German Wikipedia showed script errors. This was due to an obscure XOWA ParserFunction bug that returned ServerName as "http://de.wikipedia.org" instead of "//de.wikipedia.org"
  • Support for Wikia TabView and Tabber functions: XOWA now supports two core Wikia parser functions for generating sub-tabs within a page

(Desktop) New http://xowa.org domain with entire home wiki

I finally linked the http://xowa.org domain to https://gnosygnu.github.io/xowa . Hopefully that will make the link more memorable. Again, much gratitude goes to anonymous4 for helping me set-up the cross-domain links.

Also, I uploaded all the home documentation to http://xowa.org You can navigate to any page by taking the page url and adding http://xowa.org in front of it.

For example, http://xowa.org/ + home/wiki/Help/Contents -> http://xowa.org/home/wiki/Help/Contents

Next release: v3.4.4

I ran into a minor parser issue with Luaj that ended up pushing German Wikipedia back a week. I'll upload an update this week, along with Android wikis.

Also, Wikimedia's 2016-April English Wikipedia dump is broken. Don't bother importing your English Wikipedia this month as a lot of the pages will fail due to missing Modules. See https://lists.wikimedia.org/pipermail/xmldatadumps-l/2016-April/001298.html

For next week, I'm going to work on generating mini-wikis for download to Android. I'll also start going through some old parser bugs.


Patch: Missing wikis in sidebar (2016-04-13 21:34 Wed)

This is a patch for a defect in v3.4.2.1.

v3.4.2.1 updated the CSS for the home wiki to fix a narrow search box. See https://github.com/gnosygnu/xowa/issues/54

Unfortunately, this CSS also removed the wiki sidebar in the home wiki. I didn't notice this until yesterday.

The patch updates one file: /xowa/bin/any/xowa/cfg/app/xowa.gfs . Just replace your version with it.

You can also change this manually by finding the div_logo snippet and updating with the snippet below.

This will be part of the next release. I didn't want to release an entire new version for a one-line text-file fix.

Thanks. 

  div_logo =
<:["
  <div id='p-logo'>
    <a style="background-image: url(~{portal_logo_url});" href="~{portal_nav_main_href}" class='xowa-hover-off'~{<>msgs.get_html_accesskey_and_title('p-logo');<>}></a>
  </div>
  <div class="portal"><h3></h3></div><!-- empty heading else wikipedia css will hide next <h3> after <div id='p-logo'>; DATE:2016-04-12-->
"]:>

Release: v3.4.2.1 (2016-04-10 21:00 Sun)

The desktop app is a minor release. It has a fix for slow search database builds and some better Wikia support

The Android app is a major release. It has new pages for importing a wiki. It is the release candidate for XOWA beta on Google Play later this week.

(Desktop) Faster search database build

As per https://github.com/gnosygnu/xowa/issues/53 William was unable to build the new search databases on his machine. I tried it on an older dev machine, and found that it could take upwards of 5 hours to complete.

I optimized some of the SQL code and reduced it to about 2 hours. That said, note the following:

  • Fails on my 32-bit Windows XP dev machine: I don't know why. The same build works on 2 Windows 7 64-bit machines, 1 openSUSE Linux 64-bit machine, and 1 Mac OS X 64-bit machine. It also works on the 32-bit JVM on the Windows 7 64-bit machine. However, the lone 32-bit Windows machine fails, and fails in a very strange way (it fails silently in release mode, and hangs in debug mode). I'm going to hope this is the non-dominant behavior, and reinvestigate at a later point in time.
  • Really benefits from an SSD: SQLite really does better with an SSD. I tried some tricks like changing PRAGMA synchronous to OFF, but this only shaves off a dozen or so minutes. In contrast, an SSD will easily knock off an hour if not more (from 2 hours to about 40 min)
  • If you want the new search, downloading from archive.org is the easiest way: There are other complexities with the search database rebuild (PageRank algorithm and the 10 GB pagelinks dump: See App/Search). As such, the easiest way is to download prebuilt copies from archive.org:

(Desktop) New English Wikipedia images update (2016-03)

I uploaded new consolidated versions of English Wikipedia images (2016-03) to archive.org. See https://archive.org/details/Xowa_enwiki_latest.

In terms of usage, pick one of the following:

  • New user: Download the new 2016-03 version. Previously you would have to download 2015-06 set, and then download separate updates for every month between 2015-07 and 2016-03. This was just too cumbersome.
  • Existing user (2015-06): Ignore the 2016-03 version. You downloaded the 2015-06 set and presumably have been keeping up with the monthly updates. The 2016-03 set is the same as your set, except it is smaller (it no longer has deleted images). If you want to remove about 8 GB of deleted images, see Dashboard/Files/Deletion. Once you remove the deleted images, you will have a set that is exactly the same as the 2016-03 version.
  • Existing user (2013-11 or something else): Download the 2016-03 version. You have an old set of XOWA image databases that I really can't support any longer (they have an extension of .sqlite3). I know downloading 80 GB is painful, but I'm not planning to change database formats again in the future.

Finally, after I release XOWA Beta for Android, I'm going to work on a new update feature that will more gracefully handle monthly updates. In the meantime, please bear with the manual process.

(Desktop) Minor Wikia parser tags

A user requested some parser functionality for Wikia wikis (see https://en.wikipedia.org/wiki/User_talk:Gnosygnu . This release now supports the <choose> and <tabber> extensions.

(Android) New pages for "Import new wiki" and "XOWA Wikis"

The last major piece for XOWA Beta was more usable screens for importing wikis and managing wikis. Both these pages are now part of the latest release:

  • XOWA Wikis: The "XOWA" button in the left-nav now opens a page that lists all wikis. It also allows you to import a new one by pressing the "+" icon.
  • Import new wikis: The "Import" screen got a new design overhaul to make it much prettier and more intuitive. A lot of thanks goes to anonymous4 for helping with layout and feedback.

(Android) Minor fixes for crashes / obsoleted items

There were two minor crashes that would occur when using the left-nav button. The main one was pressing "Random" when on the "Import" screen

I also removed two features: "Nearby" and "Log in to Wikipedia". Neither of these seem useful for offline XOWA, and I didn't want them taking up screen real-estate.

Next release: v3.4.3

I'm planning on releasing XOWA Android Beta on Google Play mid-week. I'm sure that there will be spillover work that leads to another Android app update next week.

For wikis, the Wikimedia dump server is stalled again. I'm going to wait for it to resume before continuing with German Wikipedia.


Release: v3.4.1.1 (2016-04-04 20:00 Sun)

The desktop app is a minor release. It has a few fixes related to old .sqlite3 wikis and offline imports.

The Android app has not been changed.

(Desktop) Fixes for old .sqlite3 wikis

As reported in https://sourceforge.net/p/xowa/discussion/defects/thread/f142ca23/ v3.3.4 did not work for old .sqlite3 wikis. These wikis were phased out last year (in 2015-04), but I didn't want to drop support for them entirely.

v3.4.1 has 2 fixes for the following:

  • Open old .sqlite3 wikis: Previously, XOWA would just show a blank page
  • Allow old .sqlite3 wikis to upgrade to the new search system: Previously, XOWA would keep prompting to upgrade

(Desktop) Fix for offline import of large wikis

This was reported in a separate ticket: https://github.com/gnosygnu/xowa/issues/51 Basically, the new search system caused offline imports to fail for large wikis. A large wiki is defined as one with a dump file of 1.5 GB.

v3.4.1 fixes this issue. I tested with English Wikisource 2016-03: http://dumps.wikimedia.org/enwikisource/20160305/

(Desktop) Command to shrink English Wikipedia images by 8 GB

The total size of English Wikipedia is about 90 GB. 8 GB is from old images: images removed from articles during the day-to-day editing churn.

If you have English Wikipedia and have been following along with the updates, go to Dashboard/Files/Deletion and follow the instructions therein.

(Package) Release of French Wikipedia 2016-03

I had a request to update French Wikipedia, but have been delaying it for several weeks. I updated French Wiki and uploaded it here: https://archive.org/details/Xowa_frwiki_latest

Next release: v3.4.2

I'm planning to work on Android-only matters and release a "beta" app on Google play. For wikis, I'll update German Wikipedia.


Release: NONE (2016-04-03 22:00 Sun)

Sory all. I'm postponing the release for tomorrow night. I'll post again tomorrow evening. Thanks.

Namespaces

XOWA

Getting started

Android

Help

Blog

Donate