Blog/2016-09

Release: v3.9.4.2 (2016-09-26 20:00 Mon)

The PC version is a patch release. It fixes Categories not showing for pages with more than one word in the title.

Show Catgories for pages with more than one word in the title

As per https://github.com/gnosygnu/xowa/issues/81#issuecomment-249666435 Categories would not show for certain pages. It turned out this was a careless bug which affected pages with more than one word in the title.

Thanks to Ope30 for catching this bug. There's also another one he caught regarding large categories. This will be fixed in a future release or by manually downloading patched databases. See https://github.com/gnosygnu/xowa/issues/81

Release: v3.9.4.1 (2016-09-25 20:00 Sun)

The PC version is a major release. It has more Category fixes, more page sync changes, and a few other changes.

The Android version is a minor release. It has a fix for bad file downloads in Download Central.

(Wiki) Release 2016-09 French, English and German Wikipedia

I published the French wikis last week, but I'm just listing them formally in this release.

The English and German wikis were published today. To get them, do the following:

• Go to home/wiki/Special:XowaDownloadCentral
• (if you have v3.9.3.1 or earlier) Click the "Download" icon to the right of the "Available" table
• Look for 2016-09 entries in the "Available" table.

Thanks to fake4d for pointing out the bad link before.

(PC) More minor Category fixes including large categories, dynamic page list and wikidata.org

v3.9.2.1 introduced a new Category system to support Categories in HTML dump mode. Unfortunately, a few items were left out.

• Maximum of 200 pages: Some categories have more than 200 pages but XOWA would only show the first 200.
• Dynamic Page List did not work: Wikibooks, Wikiversity and Wikinews use an extension called Dynamic Page List which relies heavily on Categories. These pages always came up blank.
• Wikidata import failed: Importing wikidata.org through Import Online / Offline would fail

All of these are fixed in v3.9.4.1. In addition, I also added a feature to disable "Previous 200" on the first page and "Next 200" on the last page.

Finally, these all work with both the Wikitext dumps and the HTML dumps. I still have to get them to work on Android.

(PC) Page sync shows math and video images

v3.9.2.1 added a page sync feature, but it didn't handle Math, video and pdf images. v3.9.4.1 now handles these. There are still a few other image types to handle (helper buttons / score), but sync should now work fine for 99% of pages.

(PC / Android) Download Central does not loop endlessly if wrong file size downloaded

This is a bug that should hopefully not have affected anyone. While testing the downloads for English Wikipedia, I noticed that a "bad file size" error would basically cause Download Central to cycle endlessly. The only option would be to remove the entire task and resume.

(PC) Show pages in MediaWiki namespace

v3.9.4.1 now is able to show wikitext pages in the MediaWiki namespace. For example, https://en.wikipedia.org/wiki/MediaWiki:Edittools .

(PC) Run GUI on Raspberry Pi

As per https://github.com/gnosygnu/xowa/issues/85 the XOWA GUI was not able to run on the Raspberry Pi. I changed SWT to use WebKit in order to get this working. So far, I can only confirm that it works on my device, but hopefully it will work for others.

In addition, this build has the official release candidate sqlite-jdbc jar as per https://github.com/xerial/sqlite-jdbc/issues/58

Many thanks to nstents for helping me test the builds.

(PC) More parser fixes including Scribunto regex at end of string and gaps in argument keys

There were two significant fixes worth mentioning:

• End-of-string regexes did not work: Due to a one-off array bug, XOWA wouldn't correctly process end-of-string regular expressions in Lua. So, something like gsub( "e?$", "e", 1 ) returned wrong results
• Gaps in argument keys would fail: Scribunto allows integer keys, but with gaps. For example, something like {1="a", 4="d", 7="g"}. XOWA had logic to handle 1 gap, but failed if there were more

Next release: v3.10.1

• Wikis: After a busy two weeks, I'm going to take a break this week. If there's a wiki you want, please feel free to request!
• Dynamic update of wiki pages: I still have a little work to do on this feature.
• Image shrink: I tested a Download Central version of the image shrink but it felt too slow. I'm going to opt instead for the standalone page. I'll upload the file this week and post again in the blog.

Release: NONE (2016-09-18 22:00 Sun)

I'm skipping the release this week. I'm working on a few changes to categories, and English Wikipedia is spilling over a little.

I published French Wikipedia over the weekend. You can download them automatically with Download Central, or manually at https://archive.org/download/Xowa_frwiki_latest

English Wikipedia should go up over the next few days. German Wikipedia will follow afterwards.

Thanks.

Release: v3.9.2.1 (2016-09-11 19:00 Sun)

The PC version is a major release. It has a fix for broken Category pages, a new Category system, a new page sync feature, and other changes.

The Android version is a minor release. It has a fix for dropped connections in Download Central.

(PC) Fix for Category pages not working

v3.8.5.1 had a bug that broke Category pages. For example, https://simple.wikipedia.org/wiki/Category:Earth . This would only retrieve one random category. v3.9.2.1 fixes the issue.

(PC) New Category system to handle HTML dumps

As per https://github.com/gnosygnu/xowa/issues/81 Category pages were not showing for HTML dumps. I rewrote the Category system to handle this. Here are the high-level points:

• Works for future HTML dumps: I will be uploading new versions of 2016-09 English, German and French Wikipedia this week that will use the Category system. Unfortunately, existing HTML dumps won't work with them.
• Smaller size: The new Category system makes some database changes to reduce file size. For English Wikipedia, that means a difference between 10 GB and 8 GB. 8 GB may sound like a lot for Categories, but keep in mind there are over 100 million page to category links.
• Backward compatible: The existing Category system will work with the new code-base.
• Does not work for text database dumps: XOWA originally started off storing files in text files instead of sqlite files. I switched over to SQLite three years ago and phased out text databases two years ago. It's possible that some users with old wikis (3 years old) may still have these text databases. If so, then the new Category system won't work.

In short, the changes should be transparent to most, except HTML dump users will soon get Categories.

There's a little more info at Category Overview for those interested.

(PC) Basic page synchronization feature to get latest from online

As per https://github.com/gnosygnu/xowa/issues/72 XOWA will now be able to sync pages directly from the online version. This feature is still a work-in-progress, and probably should only be done for selective pages. Certain pages won't sync correctly for a few more releases -- namely, pages with Math. See App/Xtn/Page_sync for more info.

(PC / Android) Download Central changes for dropped connections / Mac OS X

As per https://github.com/gnosygnu/xowa/issues/80 dropped connections in Download Central would appear to hang on Mac OS X. This release should now force Mac OS X to report dropped connections. In addition, it also will try to reconnect for a minute once a connection is dropped, before giving up.

(PC) Various parser changes including Protectiong Banner, stack overflow, and invalid UTF-8

This release has a bunch of parser changes. The Change log has a complete list. The most relevant items are the following:

• Protection banners now appear: Previously, they didn't show due to a bug in returning Protection data. Currently, they will show whenever the pages says they are protected. Note that no page is ever protected from editing in XOWA: you can always click on the Edit button.
• Stack overflow error: A handful of pages would crash the program with stack overflow. These will now just show with omitted content
• Xowa/Luaj issue with foreign characters: There was a multi-threading issue with foreign characters. These are random and very low frequency. They will be fixed for future dumps.

(PC) Other minor changes including read-only support, torrent-file links and docs for HTML dump

Thanks to multiple users, this build also resolves some minor issues:

• Read-only Sqlite files are slow (thanks to Ope30): Previously, a sqlite file marked as read-only would be much slower. This build now handles read-only files correctly.
• Torrent links for non-Wikipedia wikis were wrong (thanks to malakov1): Download Central was reporting the wrong torrent links for non-Wikipedia wikis. For example, English Wiktionary was being listed as http://archive.org/Xowa_enwiktionary_latest . This build corrects the issue.
• Wrong documentation for HTML dumps (thanks to Rüdiger Gleim): The command-line script for generating HTML dumps have changed over the past few releases. I forgot to update the documentation page at Dev/Command-line/Thumbs.

Also, at risk of omission, additional thanks to markusmaresch for reporting the Mac OS X issue, and MDE186 for testing the page sync feature.

Next release: v3.9.3

• Wikis: This is going to be a busy week. I'm going to generate 2016-09 versions of English, German, and French Wikipedia with the new Category System and Random.
• Dynamic update of wiki pages: I still have more work to do on this feature. In particular, I'd like to get Math and other images showing correctly.
• Image shrink: I'm going to try to shrink the English Wikipedia images on a quarterly basis. I'm hoping to get this integrated directly into Download Central, so Android users can get it to work

Wikis: Czech Wikis (2016-09) (2016-09-05 23:00 Mon)

The Czech wikis are available in Download Central. To get them, do the following:

• Go to home/wiki/Special:XowaDownloadCentral
• Click the "Download" icon to the right of the "Available" table
• The Czech wikis will appear at the bottom of the "Available" table.

Release: NONE (2016-09-04 22:00 Sun)

This week has no release. I'm working on a variety of items based on issues at https://github.com/gnosygnu/xowa/issues Thanks to Ope30, markusmarech, and some anonymous others for all their input!

The Czech wikis are done and being uploaded now. They should be ready by tomorrow in Download Central (click the Download icon to get a new list of wikis)

For wikis, I'm going to try to update 2016-09 English Wikipedia, but I want to work on adding Categories as well as a few other items.