Blog/2016-03

From XOWA: the free, open-source, offline wiki application

Release: v3.3.4.1 (2016-03-27 19:30 Sun)

The desktop app is a major release. It has a new autocomplete address bar, a new search system as well as minor parser fixes.

The Android app is a major release. It has more updates for the new search system.

(Desktop) New autocomplete address bar

The address bar now supports autocompletion. For example, open up English Wikipedia, and start typing earth. The autocomplete is very fast and supports a lot of advanced functionality. For example you can do a search like earth + (history, future) -middle directly in the address bar. See App/Search as well as Options/Search suggest.

Unfortunately, the autocomplete uses the new search system, so old wikis need to upgrade the search databases. You can upgrade by any of the following:

(Desktop) New search system

I completely rewrote the search system over the past several weeks. Most of these changes were meant for Android to handle fast searches with a slower processor. This new engine is now part of the desktop app, so desktop users get the same benefits as well. To summarize, here are the main benefits:

  • Faster searches: Searches are now dramatically faster for single-words (earth), wildcards (e*), multiple words (earth history) and multiple wildcards (earth* history*)
  • PageRank score: Searches now use PageRank to order results. These provide better quality results than using page length. The old system would show "List of" pages at the top of the page
  • Punctuation friendly: Searches can be done for U.S.A. or USA; dont't or dont; half-life or half or life.
  • Boolean searches: Searches can use boolean operators like -, ,, +, (), and *. In addition, there's even an escape operator: \. See App/Search
  • Home wiki: The home wiki is now searchable. Try entering in search or any other word

(Desktop) Minor fixes for Category pages and taxonomy url links

There are basically two fixes:

(Android) Improvements for search system

The Android search system received more performance and accuracy improvements. Also, the Android version supports boolean searches as well. If you're using the new search system, you should definitely update your copy:

If you haven't tried XOWA on Android, please give it a try. See https://github.com/gnosygnu/xowa/wiki/Android.

Next release: v3.4.1

I've spent a lot of time on the search system for the past several weeks. I'm going to take a break and work on the following items for next week's release:

  • Shrink the image databases for English Wikipedia : English Wikipedia is getting close to 128 GB but there are about 25 GB of unused image files. I'm planning on adding a feature that allows users to delete these old images. I'll also upload a brand new set.
  • Resume image dumps: I'm going to upload a new French Wikipedia.
  • Make minor Android changes: There are still some minor changes I need to make for the upcoming Google Play release in April.

Release: NONE (2016-03-20 22:00 Sun)

I'm postponing this week's release. I'm almost done with the desktop integration of the new search system, but I still want to make a few more changes and feel that a few more days would help.

I did upload new updates for English Wikipedia this week. See https://archive.org/download/Xowa_enwiki_latest and the 2016-03-05 files. I'll update the links later to point to them.

I'll try to upload a new Android release later this week. This will mostly have small tweaks for the new search system.

Release: v3.3.2.1 (Andriod only) (2016-03-13 22:00 Sun)

Again, there is no new release for the desktop app. I will make one next week, and resume wiki generation then.

The Android app is a major release. The major change is an improved search system. To get it, do the following:

Here is a list of the changes:

  • "Wikipedia Alpha" updates no longer replaces XOWA: XOWA is a fork of the Wikipedia app. I haven't changed all the Wikipedia branding and IDs. In this particular case, I had left the package name the same. Whenever "Wikipedia Alpha" pushed out a new update, Android would prompt the user to install it. Installing it would remove XOWA and replace it with the "Wikipedia Alpha".
  • Better performance for multi-word searches: I had to add more columns and indexes to support faster lookup for certain types of searches. For example, "earth a". Unfortunately this required updating the core English Wikipedia files again
  • Do not scroll to top of results when adding new results: XOWA works by finding 2 - 5 results at a time and sending them incrementally to the search window. Each send would cause the search results to scroll back to the top. This was annoying when browsing through the list while search is running.
  • Return 10 results by default and scroll to retrieve more results: Previously, XOWA would only return 50 results. Now, XOWA is smart enough to search for more as needed. Also, performance is improved by returning 10 instead of 50.
  • Return search text as first result: XOWA now uses "page rank" to sort the results. However, most times, the search term may not be the highest ranking. For example, searching for "ear" would list "earth" over "ear" because "earth" has a higher rank. Now, XOWA will list "ear" first since that is the current search text
  • Identify redirect articles with "→": Wikipedia uses redirects to handle items like "page renames and "misspellings". For example, "Www" redirects to "World Wide Web". XOWA better identifies these entries by showing "Www → World Wide Web"
  • Remove redundant redirect articles from results: Ordinarily, there can be many redirects leading to the same page. For example, "World wide web" and "World-wide web" are both redirects to "World Wide Web". Previously, searching for "world wide web" would return all three pages. Now only 1 is returned.
  • Search for entire word by putting space at end of search: XOWA automatically adds a wildcard to every search. For example, searching for "earth" actually searches for "earth*" which will return "Earth" and "Earthquake". However, sometimes only "Earth" is wanted, not "Earthquake". Adding a space and searching for "earth " will automatically limit the search to "Earth"
  • Highlight search terms in search results: XOWA was previously relying on Wikipedia code to highlight search terms. This approach was rather simple: it would highlight a term only if the exact search string was found in the text. For example, "Earth History" would correctly highlight "Museum of Earth History" but not "History of Earth". This version now highlights the second as "History of Earth"

Again, I strongly recommend you get this version. There are a few bugs to work out, but this version is much improved over previous ones. This is the last major update for Android search, and future changes will probably not have as much of a noticeable impact -- at least until I get around to full-text search.

Release: v3.3.1.1 (Andriod only) (2016-03-06 22:00 Sun)

There is no new release for the desktop app. I'm still making major changes to the Search system and need more time for review.

The Android app is a major release. The major change is a faster search system. To get it, do the following:

Here is a brief summary of the advantages:

  • Better performance: Search is much quicker. One word searches should come back subsecond and multiple words should come back in a few, with initial results coming back in the first second
  • Better ranking: Search now ranks results by page rank. Previously, XOWA ranks by page length, which resulted in long "List of" articles appearing near the top.
  • Better search terms: Search now handles punctuation better. Searches can be done for "U.S.A." or "USA"; "don't" or "dont"; "half-life" or "half" or "life".

I'll explain these in greater detail in the next desktop release.

If you are using the Android beta, I strongly recommend you get this version. The new search is dramatically improved and well worth the effort.

Namespaces

XOWA

Getting started

Android

Help

Blog

Donate