Blog/2016-08

From XOWA: the free, open-source, offline wiki application

Blog links

2019

2018

2017


Archives

Release: v3.8.5.1 (2016-08-29 21:00 Sun)

The PC version is a major release. It adds better html database support for certain images, handles the lsth parser function, and makes minor changes to the Download Central Info page

The Android version is a major release. It also includes the same better support for certain images.

(Wiki) Publish 2016-08 English Wikipedia and German Wikipedia

The 2016-08 versions of both wikis are finally up. You can get them through Download Central.

(PC / Android) Better html database support for packed galleries, video thumbnails, and image maps

There was a small percentage (< .1%) of images that didn't appear in the HTML databases. These include the following:

The fix for the first two items required a major change to the HTML database format. The following is a brief (though technical) summary:

  • Wikitext metadata is no longer embededded with the images; previously an HTML attribute was embedded with this info. For example, <img data-xowa-image="0|220|110|-1|-1".
  • All orig_image info is early-resolved at HTML-generation time; previously, orig_image info was late-resolved at HTML-view time.
  • All file-related HTML is now correctly generated at HTML-generation time; previously, it was "guessed" at HTML-generation time. Note that this "guess" was 99.9% correct, but failed for things like "a .ogg file which is actually a video"

From a general user standpoint, this change should be entirely transparent. v3.8.5 will work with both the old HTML databases as well as the new ones. However, any version earlier v3.8.4 and earlier will not work with the new HTML databases.

Finally, the 2016-08 English and German Wikipedia are the first wikis to use the multi-threaded builder. This allowed me to cut down the English Wikipedia generation time from 80 hours to 12 hours (with 2 high-grade 8-core machines). This was a significant under-the-cover change, but hopefully most of the kinks have been worked out.

(PC) Support the lsth parser function

I discovered that a number of English Wikipedia pages used the {{#lsth}} function. In brief, this function includes a section from another page on the current page. For example: https://en.wikipedia.org/wiki/10s_BC . I ended up rewriting the lst functions to support this behavior.

(PC) Add torrent link / MD5 info to the Download Central Info page

Based on some welcome feedback from malakov1, I added a torrent link to the Download Central Info page. For example, Special:XowaDownloadCentralInfo?task id=96. Note that these torrent links are to the full archive.org folder, not the task. In particular, the following two types will be included:

  • Old database dumps: I keep at least one old set around. Otherwise, users who are downloading an old set will be suddenly cut off when a new set is published.
  • Other wikis: All wikis for a language are saved in a given folder. For example, https://archive.org/download/Xowa_dewiki_latest/ has German Wikipedia, but also German Wiktionary, German Wikisource, etc.

In most cases, you will need to manually exclude files from the torrent. Most torrent programs support this feature. For example, in qBittorrent do the following:

In addition, the Download Central Info page now includes MD5 and file size info.

Next release: v3.9.1

  • Wikis: I'm going to keep this week light and work on the Czech wikis..
  • Dynamic update of wiki pages: A primitive version of this functionality is included in this release, but it's disabled by default. If you want to try it, I'll post more information at https://github.com/gnosygnu/xowa/issues/72 . Otherwise, I plan to finish it up for the next release.

Release: POSTPONED (2016-08-28 22:00 Sun)

I'm postponing the release till tomorrow night. I uploaded 2016-08 English and German Wikipedia, but there are a few minor tweaks I want to make before release.

Thanks.

Release: NONE (2016-08-21 20:00 Sun)

I'm skipping the release this week. I'm still working on the 2016-08 English Wikipedia. I am close, but I still don't have a clean HTML dump yet. Please check again next week.

Release: v3.8.3.1 (2016-08-14 20:00 Sun)

The PC version has no release.

The Android version has a trivial release to fix an uncommon crash when starting XOWA.

(Android) Fix app crash when starting XOWA: Unable to start activity ComponentInfo{org.xowa/org.wikipedia.page.PageActivity}

Some quick details:

  • I've received two reports of this crash, but haven't been able to reproduce it.
  • The crash would occur when XOWA is starting. I think it has to do with resuming XOWA after Android has slept.
  • Starting XOWA a second time would work without a problem.

For this version, I added better guards against null references around the cited code. Thanks to the anonymous users who sent me the crash reports

Next release: v3.8.4

  • Wikis: I didn't finish 2016-08 English Wikipedia this week. I'm making a lot of significant changes to handle several minor issues. They primarily affect packed galleries, image maps, and certain video files (particularly .ogg ones). I should be on schedule for releasing 2016-08 English Wikipedia for next Monday's release.
  • Dynamic update of wiki pages: I didn't do anything on this issue this week as English Wikipedia took all my time. I'll put aside time for this week.

Release: v3.8.2.1 (2016-08-07 19:30 Sun)

The PC version is a major release. It fixes random "SSL peer shut down incorrectly" when downloading, "Script Error:Module Protection expiry" on many en.w pages, and some parser issues on pl.w.

The Android version has no release.

(Wiki) Publish 2016-07 Polish Wiktionary, Wikisource, Wikiquote, Wikibooks, Wikiversity, and Wikinews

The 2016-07 versions of the Polish wikis are up. You can get them through Download Central.

(PC) Fix random "SSL peer shut down incorrectly" when downloading database dumps

In the past week, XOWA would fail while downloading database dumps from the Wikimedia servers at https://dumps.wikimedia.org/backup-index.html . After some investigation, it looks like Wikimedia is currently resetting all connections at the 17 and 47 minute mark of every hour (EX: 1:17, 1:47, 2:17, 2:47, etc.) . For more info, see https://lists.wikimedia.org/pipermail/xmldatadumps-l/2016-August/001325.html

As the issue is still ongoing, I'm going to point XOWA to default to http://your.org instead. Note that your.org is an official backup as per https://meta.wikimedia.org/wiki/Mirroring_Wikimedia_project_XML_dumps .

If you want to change your.org to point elsewhere, do the following:

  • Go to home/wiki/Options/Import
  • Change "Dump servers" to the following 
https://dumps.wikimedia.org/,
http://dumps.wikimedia.your.org/,
http://wikipedia.c3sl.ufpr.br/,
http://ftp.fi.muni.cz/pub/wikimedia/
  • Press "Save"

Many thanks to Kevin Day and your.org for their assistance and support with this issue.

(PC) Fix "Script error:Module:Effective protection expiry" for many English Wikipedia pages

The 2016-07-20 version of English Wikipedia would show "Script error" on many pages. This was due to a minor typo on my side for "infinite" vs "infinity". It is fixed with of this release.

Thanks to William to reporting this issue.

(PC) Fix parser issues for Polish Wikipedia pages, including location maps and broken infoboxes

Polish Wikipedia required two major changes:

  • Location maps weren't showing due to issues with JSON parsing.
  • Some infoboxes were garbled because of incorrect parsing of <ref%> tags.

Next release: v3.8.3

  • Wikis: I'm working on 2016-08 English Wikipedia now, and want to add a few fixes for this HTML dump release. Unfortunately, the new HTML generator is not ready yet, so English Wikipedia will probably take the full week.
  • Dynamic update of wiki pages: I added "wikimedia API" calls this week. I'm going to work on recursively parsing all templates for this week.requested in https://github.com/gnosygnu/xowa/issues/72 . This will probably take a few weeks. For next week, I'm going to work on the simpler "wikimedia API" calls
  • Redlinks for HTTP Server: This has kept slipping for a few weeks. I'm going to table this for now, and come back to it later.
  • mwad integration with the status bar: I'm also tabling this as well.

Release: v3.8.1.1 (2016-07-31 19:30 Sun)

The PC version is a minor release. It fixes a bug with "invalid title" when clicking on the left-hand sidebar.

The Android version is a minor release. It fixes a random crash when restarting the app while downloading wikis

(Wiki) Publish 2016-07 Arabic Wiktionary, Wikisource, Wikiquote, Wikibooks, Wikiversity, and Wikinews

The 2016-07 versions of the Arabic wikis are up. You can get them through Download Central.

(PC) Fix "invalid title" when clicking on wikis in left-hand nav

This was kindly reported by markusmaresch in https://github.com/gnosygnu/xowa/issues/73 . Basically, clicking on a wiki in the left-hand sidebar would fail with "invalid title". This was due to some changes I made with the UI in a recent release.

v3.8.1 fixes the issue

(PC) Minor wikibase update for "renderSnaks"

Wikidata (www.wikidata.org), or "Wikibase", is a continually evolving project. French Wikipedia started using two new methods recently: renderSnaks and renderSnak. I added support for these calls in this release. I also cleaned up some Wikibase code.

(Android) Fix random failure when restarting XOWA while downloading wikis

A helpful anonymous user forwarded me a crash report for XOWA Android. Sometimes, XOWA Android would fail to restart when in the process of downloading wikis from Download Central. This was broken since the initial release of Download Central.

This fix was pushed out by a separate Play Store release last week. It is included formally in the apk-build now.

Next release: v3.8.2

  • Wikis: I'm going to work on Polish this week.
  • New HTML database generator: I'm still been plugging away at the HTML generator. It's closer to 99% now, and I'll start phasing it in over the next few releases.
  • Dynamic update of wiki pages: This was requested in https://github.com/gnosygnu/xowa/issues/72 . This will probably take a few weeks. For next week, I'm going to work on the simpler "wikimedia API" calls.
  • Redlinks for HTTP Server: This too has also slipped another week.
  • mwad integration with the status bar: Another week of slippage here as well.

Namespaces

XOWA

Getting started

Android

Help

Blog

Donate