Blog/2016-01

From XOWA: the free, open-source, offline wiki application

Blog links

2019

2018

2017


Archives

Release: v3.1.5.1 (2016-01-31 22:00 Sun)

The desktop app is a major release. It resolves reference errors in 2016-01 English Wikipedia, fixes wide images on German Wikipedia, and adds a new PageBanner extension

The Android app is also a major release. It shows more tables, heiroglyphics, and images with a ? in title.

Luaj fixes for references in 2016-01-13 English Wikipedia

The 2016-01-13 dump had many erros in the reference section due to bugs in Luaj. One of the following would be displayed:

  • Check |archiveurl= value (help)
  • Specials character in |{some word}

These were bugs in Luaj with the frontier pattern and multi-byte chars. They were both resolved in this release.

Fix for German Wikipedia and extremely wide pie charts

I'm extremely thankful for this report at https://github.com/gnosygnu/xowa/issues/38 There were HTML issues in German Wikipedia because all the wikis rely implicitly on American formatting rules for decimals. For example:

XOWA was doing this multiplication using the user's locale.

  • For US users (like me), 18*.9 was calculated correctly as 16.2
  • For German users though, 18*.9 was calculated incorrectly as 162.
    • This was because Java was interpreting the .9 using German number-format rules. Since "." is the thousands separator, and "," is the decimal separator, .9 became 9. For US users, imagine ,9 becoming 9.

v3.1.5 fixes this issue by asserting that all decimal-parsing is done in the US locale.

New PageBanner extension for Wikivoyage

This was an extension introduced by Wikimedia late last year to standardize the display of the image, icons and table of contents at the top of the page. This involved some considerable work on my side (including a Mustache parser), but XOWA now reproduces the same look.

(Android) Show navboxes and other tables

The Wikipedia Mobile app deliberately hides navboxes as well as other tables. I think this is done for the following reasons:

  • These tables were meant for desktop display, and often had wide widths
  • These tables clutter the page.

In contrast, I really do like showing as much content as possible, so I decided to show these tables. I'll probably put an option to disable this later, but I'm going to wait first for some feedback from others

(Android) Always center table content

This is another deviation from the Wikipedia Mobile app style. Basically most tables are left-aligned and would end before reaching the right-hand side of the page. I changed the CSS to have these tables fill up the entire page and center the content.

This has one unfortunate sideback in that it breaks the "Collapse" buttons. I'll try to fix this in a future release, but if I can't, I may remove them entirely.

(Android) Show hieroglyphics

The early versions of the XOWA Android app did not show hieroglyphics. This is now supported in this build

(Android) Show images with ? in file title

This was a bug wherein images that had a ? were not being displayed within the Android WebView. I fixed this by replacing ? with %3F.

Next release: v3.2.1

I ran into several issues with the 2016-01 English Wikipedia dump that set me back a few days. I'm planning to upload English Wikipedia images and HTML this week, and then continue on with the other English wikis

For code, I'm going to tackle performance on the Android search again as well as a few minor UI issues on Android (collapse-table, pinch-zoom)

Release: v3.1.4.2 (2016-01-28 11:00 Thu)

This is an interim release for the desktop app. It fixes script error messages in 2016-01-13 English Wikipedia

Scribunto fixes for 2016-01-13 English Wikipedia

I accidentally had the wrong dump in place when kicking off the 2016-01-13 build. As such, I didn't spot the issue till this morning after I restarted the build.

Basically, the 2016-01-13 English Wikipedia dump introduced new page admininistration functions. These include {{PROTECTIONEXPIRY}} as well as a Scribunto function called getStabilitySettings. This resulted in a bold red "Script error" message at the top of every protected page. There are a lot of protected pages.

v3.1.4.2 resolves both issues. So far, it's stable across 35,000+ pages, but if you run across any issues, please report them. Thanks.

Release: v3.1.4.1 (2016-01-24 19:30 Sun)

The desktop app is a major release. It resolves several wiktionary errors on non-English pages and has better HTML dump generation for other wikis.

The Android app is a major release. It has a more responsive search, instant image loading when reloading page, and a fix for pages failing to load if they have spaces and colons in the title.

Scribunto / Luaj fixes for English Wiktionary

There were several dozen pages with script errors in English Wiktionary. They basically fell into these categories:

  • Luaj / Scribunto errors with MCBS entities in Asian languages (1 glyph; 2 char).
  • Programmatic errors for minor cases
    • Recursive gsub calls with lua function
    • Search position returned as int not string

The build log for English Wiktionary looks a lot cleaner now, though I still have to add some support for the {{categorytree}} function.

Better HTML dump generation for English Wiktionary, Wikisource, Wikiquote, etc. Also, French Wikipedia

I ran the HTML dump generator through all the English wikis, as well as French Wikipedia. The results looked good, but there were a lot of log errors that were addressed with this release.

(Android) More responsive search

Search is still slow on Android. I'll fix this in a future relase, but I'll have to change the format of the search databases.

In the interim, I hacked in the following approach. Assume a search term of "Earth":

  • Search for titles that match the term. For example, "Earth". This is fast and usually comes back in 0 - 3 seconds.
  • Search for titles that have the term in any word within the title. For example, "History of Earth". This is moderate and usually comes back in 5 - 10 seconds.
  • Search for titles that have the term starting in any word within the title. For example, "Earthquake". This is slow and usually takes 15 seconds and sometimes more.

(Android) Instant image loading when reloading page

First, some background. XOWA loads pages in two stages:

  • Show the HTML
  • Fill in the images one by one

The 2nd stage is necessary because it takes a few tenths of a second to retrieve an image from the database. For large pages (20+ images), this would delay page load by a few seconds (for example: 20 images * .2 second = 4 seconds).

However, once a page is viewed, reloading a page shouldn't have to re-retrieve the image from the database. The previous releases were actually re-retrieving the image. This release just uses the image if it's already there.

(Android) Fix for pages failing to load if they have spaces and colons in the title

This only affects a small minority of items that have a colon and spaces in the title. For example, "List of A:B items". XOWA didn't load this page because the underlying Wikipedia Mobile App was encoding the spaces as +.

Next release: v3.1.5

I haven't uploaded wikis this week, because I'm working on a blocking item in Wikivoyage (Pagebanner). I'm going to delay another week and focus on updating 2016-01-13 English Wikipedia. This means a new HTML dump which should be available by next Sunday. I'm also hoping to upload the other English wikis, and then continue on to French, etc.

Meanwhile, I'm going to delay the Google Play Store a few weeks. If anyone thinks I should go to Google Play Store earlier, please let me know. From my side, there are a number of items that still need polish. If you have any feedback, please do post in https://github.com/gnosygnu/xowa/issues/36 I'd appreciate anything: from "Works fine" to "This feature needs some more work".

Finally, if you have an Android device and want a wiki for your device, let me know which, and I'll bump it up in priority. The next items in queue are French, German, Chinese and Arabic Wikipedia. I'll get to all the wikis but it may take a few months unless I hear requests otherwise.

Release: v3.1.3.1 (2016-01-17 21:45 Sun)

The desktop app is a trivial release. Its main feature is a help page for generating HTML-dumps

The Android app is a major release. It has a Random feature, shows more images, and adds CSS-tweaks.

Documentation for html-dump script. See Dev/Command-line/Dumps

This item is self-explanatory. The XOWA Android app is getting more stable, so I felt it would be time to document the generation of the HTML databases.

Improved Android beta, including Random, more image display and CSS-tweaks. See https://github.com/gnosygnu/xowa/issues/36

This build has a number of new features.

  • Random now works in offline mode.
  • All thumb and gallery images should now show up. (approximately 5% of them weren't before)
  • CSS tweaks were added to center the images as well as standardize some background colors

You can get the latest release here: https://github.com/gnosygnu/xowa/releases/

Next release: v3.1.4

The Wikimedia dump service resumed, but I'd like to delay French, Polish and Latin for another week. I'm planning to generate HTML dumps for the rest of the English wikis including Wiktionary, Wikisource, Wikivoyage, etc. This will be a good stress test for the new XOWA html-dump format before I continue on to other wikis.

I'm also going to continue tweaking the XOWA Android app. If you have a moment to help out with testing, please set it up and give me some feedback: https://github.com/gnosygnu/xowa/issues/36

Release: XOWA (None); XOWA Android (v0.1.2.0) (2016-01-10 20:30 Sun)

There's no release for the XOWA app tonight. I've spent the week mostly on the Android app, and there were no real changes for the main desktop app.

I have released an Alpha of the XOWA Android App. You should check the following links:

If you have Android KitKat or higher and would like to run XOWA on it, I'd be greatly interested in feedback. See the Android progress thread for more details.

Next release: v3.1.3

I still have some tasks for XOWA Android that I'd like to work on this week. I am going to try to release the main app next week, but if there are no new changes it may slip again in favor of an update of the XOWA Android release

I haven't generated any wikis this week as the dump service seems to have come to a halt. Whenever it resumes, I'll pick up again with the promised updates of French, Polish and Greek

Release: v3.1.1.1 (2016-01-03 20:00 Sun)

v3.1.1 is a minor release. It has a few fixes for some English Wikipedia pages, and many trivial HTML changes.

Handle refs inside {{#tag}}

I only saw this issue on one page in English Wikipedia (United Kingdom), but it has potential for more effects. Basically, XOWA did not handle <ref> inside a {{#tag}}. For example: {{#tag:ref|<ref>a</ref>}}. For the United Kingdom page, this threw off the format, and made the Infobox as wide as the entire page.

Fix for incorrect search word counts and IPA links

There was a minor issue with incorrect search counts for words. I don't think this affects anything for desktop searches, but it did throw off the Android search.

Meanwhile, due to incorrect URL generation, IPA links would not download from the web after being clicked.

Trivial HTML changes

v3.1.1 has a bunch of HTML changes to make it slightly closer to MediaWiki's HTML.

  • The most significant one is the removal of the magnify icon, which is now being handled in CSS as opposed to a separate <img> tag.
  • There are a few other ones affecting incorrect escaping of quotes. This would cause incomplete titles when hovering over images or links.
  • Finally there are minor whitespace / attribute generation issues. None of these should be noticeable; they were made to handle log messages during bulk HTML generation

Next release: v3.1.2

I was on track to release the new Android version tonight, but ran into a last minute issue with thumbs and English Wikipedia. I'm pushing this out another week.

For wikis, I'll work on the French Wikipedia refresh, and try to add on Polish and Greek.

Finally, an apology to those I've promised minor changes. I've spent a lot of time on Android recently, and there's been a backlog of items. I'm hoping to be done with the Android version in the next few weeks and will work on these items as I get time.

Namespaces

XOWA

Getting started

Android

Help

Blog

Donate