Dev/Design/Offline files

This process is currently in development. The documentation below is provided for reference only. You can run the commands, but please note that the end result will not have any immediate use.

Offline thumbnails will require a lot of time and disk storage. Estimates are provided below in the format of [Space used; Time taken].

Note that Simple Wikipedia will take 24+ hours and 120 GB for the first run.

Commons process

Since most images reside in the commons wiki, commons must be set up. Note that this only needs to be done once for all wikis.

• [ 50 GB; 03 h] Import the Commons wiki with category level 2. See Dashboard/Import/Offline
• [ 50 GB; 02 h] Download and unzip commonswiki-latest-image.sql.gz
• [ 01 GB; 20 m] Run the images command. This will generate a database called oimg_image which will have more data for each file (width, height)
• [ 15 MB; 40 m] Run the redirect command. This will generate a database called oimg_redirect which will list all redirect targets

Wiki process

These steps must be run for the given wiki. Estimated size / times are provided for Simple Wikipedia.

• [ 01 MB; 01 m] Run the images command.
• [ 05 MB; 01 m] Run the redirect command.
• [ 25 MB; 15 m] Run the lnki_temp command. This will generate a database called oimg_lnki which will be the main database. It also creates a table called oimg_lnki_temp which lists every [[File:]] reference
• [ 20 MB; 05 m] Run the file_regy command. This will create a table called oimg_lnki_regy which aggregates all info in oimg_lnki_temp. It also creates a table called oimg_file_regy which uses the data generated in the images and redirect commands.
• [ 05 MB; 01 m] Run the xfer_regy command. This will create a table called oimg_xfer_regy which generates a unique list of all thumbnails to download

app.bldr.cmds {
  add     ('commons.wikimedia.org', 'oimg.image');
  add     ('commons.wikimedia.org', 'oimg.redirect')  {commit_interval = 10000; progress_interval = 100; cleanup_interval = 100;}
  add     ('simple.wikipedia.org' , 'oimg.image');
  add     ('simple.wikipedia.org' , 'oimg.redirect')  {commit_interval = 10000; progress_interval = 100; cleanup_interval = 100;}
  add     ('simple.wikipedia.org' , 'oimg.lnki_temp') {commit_interval = 10000; progress_interval = 100; cleanup_interval = 100;}
  add     ('simple.wikipedia.org' , 'oimg.file_regy');
  add     ('simple.wikipedia.org' , 'oimg.xfer_regy');
}
app.bldr.run;