Blog/2016-07

From XOWA: the free, open-source, offline wiki application

Contents

Release: v3.7.4.1 (2016-07-25 19:00 Mon)

The PC version is a minor release. It fixes a build issue when building wikis from old dumps by command-line script.

The Android version is a major release. It has a new screen to change the wiki location for Download Central

(Wiki) Publish 2016-07 English Wikipedia

The 2016-07 versions of English Wikipedia is up. You can get them through Download Central.

For images, if you want the latest:

  • If you are a current user of 2016-06 English Wikipedia, you only need to download the 2016-07 Image Update to get the latest images.
  • If you are a brand new user, you will need to download 2016-06 and 2016-07 Images

For articles, if you want the latest:

  • Regardless of whether or not you're a current user or a brand new user, you will need to download the entire 2016-07 Articles. There is no incremental version of the articles dump, though I'd like to have one up by end of the year.

(Android) Add screen to change wiki location for Download Central

First, some background. XOWA Android tries to use the external SD card for downloading wikis. Unfortunately, this is not easy, as different devices may have different paths for the external SD card (it varies by Android version as well as by manufacturer). If XOWA doesn't guess correctly, then the wikis will be downloaded to internal memory

In v3.7.1, I added some logic to try to correctly detect the external SD card for one device (BLU on Android Lollipop). In the process, I ended up making the detection worse for other devices.

v3.7.4 attempts to fix this by reverting back to the previous detection strategy. In addition, I also included a new "folder selection" screen to manually choose the external SD card.

You can access this folder selection screen by clicking the "folder" icon in Download Central. Note that you will need to restart the app after you change the folder.

Note that you can always manually download the wiki from archive.org and import them using the "XOWA file browser". The "XOWA file browser" is available through the "folder" icon on the "XOWA" Wikis page.

(PC) Fix automatic deletion of pagelinks file

This was a minor bug with building wikis by command-line import for historical wikis (for example, English Wikipedia from 2016-01).

XOWA would automatically delete all .gz files after the wiki was built but before the search score was built. This deletion was part of some over-zelous cleanup which deleted all .gz files, not just those that have been used. The .gz-deletion crashed the command-line build because it caused the pagelinks file to be deleted before it was used.

I never caught this because I always work with the current dumps. For current dumps, the XOWA command-line script has a command (util.download) which automatically unzips the .gz file to .sql. The deleted .gz file didn't affect the build because the .sql file was available.

v3.7.4 now only deletes .gz file after it is used.

Next release: v3.8.1

  • Wikis: I'm going to do Arabic this week.
  • New HTML database generator: I've still been working on the HTML generator. It's about 95% there, but the last 5% is pretty hard. I'm hoping it will be ready for 2016-08 English Wikipedia
  • mwad support: This has slipped another week.
  • Redlinks for HTTP Server: This too has also slipped another week.

Release: POSTPONED (2016-07-24 23:00 Sun)

I'm postponing the release till tomorrow night. I'll post again then. Thanks.

Release: v3.7.3.1 (2016-07-17 20:30 Sun)

The PC version is a major release. It fixes redirects for HTML databases and minor TOC issues.

The Android version has no release.

(Wiki) Publish 2016-07 French Wikipedia, Wiktionary, Wikisource, Wikivoyage, Wikiquote, Wikibooks, Wikiversity, and Wikinews

The 2016-07 versions of the French Wikis are up. You can get them through Download Central.

(PC) Fix Redirects not working for HTML databases

I discovered that HTML databases didn't support redirects. So, if page "A" was a redirect to page "B", going to page "A" opened up a blank page.

With v3.7.3.1, going to page "A" now redirects correctly to page "B".

(PC) Fix minor Table of Contents issues for Wikitext / HTML databases

There were a number of minor Table of Contents (TOC) issues where clicking on the TOC would not scroll down to the correct header section.

They fell into 3 categories:

  • TOC items with a <br/>
  • TOC items wherein two were a duplicate and a third was a duplicate with a " 2" suffix. For exampe: 2 sections both titled "Example" and 1 section titled "Example 2"
  • TOC items with a reference or other structural html (list; table; etc)

There is also a notable entry for Wikivoyage Pagebanner items which also generates a TOC, but embedded in the page banner image.

The fix for all this turned out to be surprisingly complicated and took up a good deal of time this week. They will only work for all Wikitext wikis, and all HTML wikis generated after this week.

Note that aside from the Wikivoyage item, the other TOC items probably affect < .1% of all pages.

Next release: v3.7.4

  • Wikis: I think 2016-07 English Wikipedia is done, and will be proofing it this week. I'll also try to start Arabic.
  • New HTML database generator: I've been working on this for about two weeks. Basically, I'm rewriting the HTML database process so that it supports multiple-threads. I'm planning to test generating wikis with the new process this week, though I think I may not fully replace wiki generation for another 2 weeks.
  • mwad support: I didn't get a chance to look at status-bar integration this week. It's been pushed to this week.
  • Redlinks for HTTP Server: This too has also slipped a week. Next week looks doubtful as well.


Release: v3.7.2.1 (2016-07-10 21:00 Sun)

The PC version is a major release. It fixes a breaking change for importing 2016-07 wikis. It also adds Table of Contents for HTML databases, fixes an HTTP server bug, and includes mwad -- a new utility for creating XML dumps

The Android version is a minor release. It handles the new 2016-07 wikis and the Table of Contents support.

(Wiki) Publish 2016-07 Haitian Wikipedia

I fell behind on French wikis this week while waiting for the new month's dumps. French Wikipedia is uploading now, and the rest will be generated and uploaded during the week.

In the meantime, I added Haitian Wikipedia as per the request at https://github.com/gnosygnu/xowa/issues/67

(PC) Fix 2016-07 English Wikipedia import failure

2016-07 English Wikipedia failed to import. This was due to the categorylinks.sql file and some specific SQL parsing in XOWA.

v3.7.2.1 resolves this issue, and English Wikipedia now imports correctly

(Server) Fix images not showing for Wikitext databases

This was kindly reported by ve3lst in https://github.com/gnosygnu/xowa/issues/68 I broke images for Wikitext databases back in v3.6.4.3. Note that images do show for HTML databases (databases from Download Central).

v3.7.2.1 resolves this issue. Viewing a page on the HTTP Server will now shows images.

(PC) Add Table of Contents support for HTML databases on the PC

v3.7.2.1 now generates Table of Contents for HTML databases on the PC. Note that this only affects new wikis such as Haitian and French. The 2016-07 version of English Wikipedia will have them when I release it next week. I'll redo German Wikipedia later to include this as well.

(PC) Include mwad (MediaWiki API dump) for creating XML dumps for Wikia and other wikis {contributed by Mattze96}

For those who are unaware, Mattze96 was the amazing developer who contributed the original HTTP Server code. This time, Mattze96 contributed mwad: the MediaWiki API dump tool. This tool allows creating XML dumps for Wikia and other wikis (https://wiki.archlinux.org/).

If you use XOWA for Wikia wikis, or other non-Wikimedia wikis, you definitely want to try it out. For more info, see App/Import/mwad as well as https://github.com/Mattze96/mwad and https://github.com/gnosygnu/xowa/issues/66

(PC) Fix minor Wikibase issues, including wikibase-item

There are two issues here:

  • Scribunto failure due to wikibase-item: On a low level, XOWA was using the term "wikibase-entityid". Instead, it should have been "wikibase-item". This was causing an error in Scribunto that generated script errors on many ht.w pages
  • species.wikimedia.org links were not showing in the "In Other Languages" section: This is a fairly minor issue. XOWA shows "In other languages" at the bottom of the page. Due to a change in 3.6.4.1, links to species.wikimedia.org were not showing

(PC) Target Java 1.7, not Java 1.8

I noticed that I accidentally bumped the Java version requirement from 1.7 to 1.8 back in 2016-03. v3.7.2.1 now requires Java 1.7 only

Next release: v3.7.3

  • Table of contents minutiae: There are a few minor bugs for TOC. In addition, it still needs full integration with Wikivoyage. Both of these will be part of the next release
  • mwad support: I'm going to try integrating this into the XOWA status bar.
  • Wikis: I'm uploading French this week and then will shift to English Wikipedia for next week week.
  • Redlinks for HTTP Server: This slipped from this week's schedule. I'm going to keep it on the schedule for next week, but it may go another week.

Release: v3.7.1.2 (2016-07-03 23:40 Sun)

(PC & Android) Fix broken search for multiple words in 2016-05 wikis and earlier

This fix was reported in https://github.com/gnosygnu/xowa/issues/64 and https://github.com/gnosygnu/xowa/issues/65

I broke then multiple-word search function when tweaking the search algorithm to reduce the importance of shorter pages. This bug affected 2016-05 wikis and earlier. I missed it because I'm working on 2016-06 wikis.

I put a fix in now and tested with a 2016-05 copy of English Wikipedia.

Thanks to William and Ope30 for reporting these issues!

Release: v3.7.1.1 (2016-07-03 18:00 Sun)

The PC version is a minor release. It adds Gallery support for HTML dtabases.

The Android version is a minor release. It fixes a number of severe crashes for outlier situations.

(Wiki) 2016-06 German Wiktionary, Wikisource, Wikivoyage, Wikiquote, Wikibooks, Wikiversity, and Wikinews

These wikis are available at Download Central.

(PC) Support Gallery

On the PC, the HTML databaes showed Galleries vertically and with incorrect styles (no gray border box).

This build fixes some galleries for the existing 2016-06 wikis. Unfortunately, the other galleries can only be fixed in the next HTML dump: 2016-07 and beyond.

(PC & Android) More minor fixes for Download Central including connection disruption and WAKE_LOCK fixes

There are basically two types of fixes:

  • Show error when downloading: A user reported intermittent problems with download central: https://github.com/gnosygnu/xowa/issues/63 . A little later in the week, I discovered archive.org can sometimes drop a connection after a few hours. In these cases, XOWA would just stop without any indication. This version prints status messages to the screen, and also allows the user to resume
  • Android would sometimes time out: Android sometimes stops XOWA while downloading. This version allows XOWA to keep Android awake (WAKE_LOCK) until downloads finish.

(Android) Multiple fixes for severe crashes

There were a number of fixes for crashes in uncommon scenarios. Most of these involved switching SD cards, or deleting wikis outside XOWA. The crash was bad enough such that XOWA would not work unless the Storage files were cleared (or XOWA was reinstalled).

These are fixed in this version. For more info, see the Change log

Next release: v3.7.2

  • Table of contents: I did more work on the TOC, but wasn't able to integrate it for this release. It should be part of the next one.
  • Redlinks for HTTP Server: This also didn't make it into this release.
  • Wikis: I'm working on French next and adding them to Download Central.

Namespaces

XOWA

Getting started

Android

Help

Blog

Donate