Dev/Design/Database

Overview

As of v2.4.1.1, XOWA has changed the database layout to be more simple and portable. Regular users should not notice any difference. Users who move XOWA wikis between computer systems will notice some benefits.

Details

• .xowa: XOWA databases now have the extension .xowa instead of .sqlite3.
  • For example, importing Simple Wikipedia now generates a file called simple.wikipedia.org-text.xowa instead of simple.wikipedia.org.000.sqlite3
  • Note that the underlying file is still a sqlite3 database. The extension has just been renamed from .sqlite3 to .xowa.
  • Note that all new wikis uploaded to archive.org will be in .xowa format. The same also goes for new torrents.
• Backward compatible: XOWA v2.4.1.1 and higher opens both .xowa wikis and .sqlite3 wikis. There is no need to upgrade to the new wikis unless you're planning on using some of the features below. However, users are encouraged to generate new wikis with the new format.
• One file for most wikis: Small wikis will default to one file instead of several.
  • For example, importing simple.wikipedia.org will generate 1 file called simple.wikipedia.org-text.xowa. In contrast, it used to generate 5 files: simple.wikipedia.org.000.sqlite3 up to simple.wikipedia.org.004.sqlite3
• Multiple 'detachable' files for large wikis: Larger wikis will still generate multiple files. See the Multi-file layout section below for more detail.
• Databases in same directory: All database files are now under /xowa/wiki/wiki_name. File databases no longer have to go to /xowa/file/wiki_name. The latter was a point of confusion for new users, as well as an inconvenience when trying to move wikis to new machines.

Options

General behavior for "single" vs "multiple" file import can be customized at Options/Import under Database Layout

Multi-file layout

Large wikis will still generate multiple files. This includes English Wikipedia, German Wikipedia as well as about a dozen others. The primary reasons are as follows:

• FAT32 4 GB limit: FAT32 supports a maximum file size of 4 GB. Although modern operating can support files greater than 4 GB, most flash cards still use a FAT32 file-system. In addition, large files are not supported in other formats (for example, DVDs, archive.org with a 10 GB limit, etc.)
• Detachable databases: Wikis will still work even if individual databases are deleted or removed. For example, if you don't plan to read the articles in the Wikipedia namespace and want to save 2.6 GB of disk space then you can simply remove the file called "en.wikipedia.org-text-ns.004.xowa".

Note that detached databases can be reattached simply by moving the file back into its proper location. For example, consider the following:

• C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org-xtn.category.core.xowa is deleted.
  • Categories will no longer show, but the wiki functions as normal
• C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org-xtn.category.core.xowa is restored.
  • Categories will now work again.

The following table provides more information on what happens when a database is detached:

file-name suffix	example	effect if removed
core	en.wikipedia.org-core.xowa	Core file: wiki will not work
text-ns.***-db.***	en.wikipedia.org-text-ns.012.xowa	Articles in the namespace will not be available. See http://en.wikipedia.org/wiki/Wikipedia:Namespace For example, ns.012 is for "Help:" articles
xtn.category	en.wikipedia.org-xtn.category.xowa	Categories will not show up at the bottom of the article
xtn.index	en.wikipedia.org-xtn.index.xowa	"Special:Search" will not work; EX: en.wikipedia.org/wiki/Special:Search/Earth?search=Earth&fulltext=y
wbase	www.wikidata.org-wbase.xowa	Core file: Wikidata will not work.
file-core	en.wikipedia.org-file-core.xowa	Wiki will not show images
file-data	en.wikipedia.org-file-data.001.xowa	Images in that database will not show

Version compatibility

The formats for wiks and files are interchangable between old and new formats. A user may keep their existing old file databases and still generate new wikis whenever they want to.

For example, consider English Wikipedia. All of the following combinations of old and new should work:

wiki type	file type	wiki path example	file path example
old	old	C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org.000.sqlite3	C:\xowa_release\file\en.wikipedia.org\fsdb.main\fsdb.abc.sqlite3
new	old	C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org-core.xowa	C:\xowa_release\file\en.wikipedia.org\fsdb.main\fsdb.abc.sqlite3
old	new	C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org.000.sqlite3	C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org-file-core.xowa
new	new	C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org-core.xowa	C:\xowa_release\wiki\en.wikipedia.org\en.wikipedia.org-file-core.xowa