KDE Localization - l10n-stats Documentation

l10n-stats Documentation

Table of Contents
Introduction
What is L10n-stats ?
L10n-stats Configuration
- Grabbing Statistics (stats-datas.script.php)
- Crontab Example
Web Translation Statistics Configuration
- Apache Virtual Host Configuration
- Customize Web Translation Statistics Style
SQL Schema Description
Required tree of localization repository
Troubleshootings
Use a newer version
Request and Bug reports
Credits
- GUI Statistics
- Documentation Statistics
Todo list

Introduction

The goal of this project is to provide an easy preview of the localization status of a project. Generally, a project is divided in different parts, to be able to be translated by several people at the same time, in different languages.

These parts are called PO files, and these files are analyzed by l10n-stats, to display translation status of each one, for each language supported.

Of course, a web site demo will explain better than thousand words the use you can have of the l10n-stats project. Please see, for example, the KDE localization web site http://l10n.kde.org/

You should find a more updated documentation here.

The scripts use the PHP language, and will require the PHP command line to work.

What is l10n-stats ?

L10n-stats is a set of PHP scripts used with a MySQL database providing HTML pages of the localization status of a project using Gettext. This project was written initially for KDE project, and a demonstration could be found at http://l10n.kde.org

The l10n-stats scripts regurlarly update a database to see how your project is translated, by translation teams, by packages, and even by specifical branches.

It uses a local copy of any SVN repository to process PO files, saves these information to a MySQL database, and then datas are processed using PHP scripts.

The only requirement you need is to have previously organized your localization tree files like the KDE project. We will talk about this in an other paragraph. Do not be afraid, this will not be a problem if your project is correctly organized.

L10n-stats Configuration

Download a tarball of the current version of l10n-stats from the download page of this project. Uncompress it in your home folder for example, or whatever.

Then, please copy the configuration file (l10n-stats.conf) to your home directory root and rename it to .l10n-stats.conf (do not forget the first "." in the name) You can alternatively copy this file to the /etc folder : /etc/l10n-stats.conf.

For more information, you can download the default config file here : l10n-stats.conf to see how it looks like.

The parameters of the .l10n-stats.conf file in your home folder will be used by default by l10n-stats. The l10n-stats.conf file in the l10n-stats folder just contains default parameters, and they will be overriden by the one you set up in .l10n-stats.conf.

A good solution is to only keep configuration entries that you have modified in your .l10n-stats.conf file, because most of configuration entry are for advanced users, and do not need to be changed.
Create a database in your MySQL server, with the name you want. Configure the .l10n-stats.conf to allow scripts to connection to this database correctly.
Use the tables-creation.sql SQL scripts to create needed tables of l10n-stats. (through mysql command line, or phpMyAdmin if installed.)
Create a working copy of your SVN repository on the machine you're going to use l10n-stats scripts. This working copy must respect some specifical criteria, described in an other paragraph.
Use the datas-examples.sql SQL example to configure properly your project and then execute this script on your MySQL database.
Configure the different options of the .l10n-stats.conf file.

Generally, you only need to modify paths to SVN repository and login/password to access to your database.
Test your installation by launching the following script of l10n-stats project : stats-datas.script.php

Grabbing Statistics (stats-datas.script.php)

Read local working copy of a SVN repository which contains PO files of a specifical project. See the KDE SVN project to know how to order your repository (http://websvn.kde.org) The statistics are saved in a MySQL database previously created. This first task is mandatory if you want to generate pages. To see graphical history, use the Crontab to launch this script everyday.

Crontab Example

You can use the following lines to regurlarly update and regenerate your statistics page.

Use the crontab -e command and type the following lines :

############ L10n pages generation ##############
# Launch stats-datas.script.php script, everyday, at 4h o'clock
00 4 * * * cd / ; ./stats-datas.script.php

Web Translation Statistics Configuration

The PHP scripts displaying statistics pages are available here : http://websvn.kde.org/trunk/www/areas/l10n/stats

Simply checkout these scripts in the stats/ folder of your website. You will also need an .htaccess file and the media/ framework of KDE website.

So, simply execute the following commands :

mkdir -p /var/www/statistics
cd /var/www/statistics
svn export svn://anonsvn.kde.org/home/kde/trunk/www/areas/l10n/stats
svn export svn://anonsvn.kde.org/home/kde/trunk/www/areas/l10n/img
svn export svn://anonsvn.kde.org/home/kde/trunk/www/areas/l10n/menu.inc
svn export svn://anonsvn.kde.org/home/kde/trunk/www/areas/l10n/site.inc
svn export svn://anonsvn.kde.org/home/kde/trunk/www/areas/l10n/css.inc
svn export svn://anonsvn.kde.org/home/kde/trunk/www/areas/l10n/.htaccess
svn export svn://anonsvn.kde.org/home/kde/trunk/www/media

Then you need to copy the l10n.kde.org.conf Example File to the following path of your server :
```
/etc/l10n.kde.org.conf
```
Of course, you need to configure login and password to access to your database in this configuration file.

Apache Virtual Host Configuration

Please note that Web Translation Statistics needs an enabled mod_rewrite in your Apache 2.

Here is an example of a working Virtual Host configuration :

<Directory "/var/www/statistics">
       Options -Indexes +FollowSymLinks
       AllowOverride All
       Order allow,deny
       Allow from all
</Directory>

<VirtualHost *:80>
       DocumentRoot "/var/www/statistics"
       ServerName statistics.com
       ServerAdmin admin@statistics.com

       Alias /media /var/www/media

       php_value include_path ".:/var/www/media/includes:/usr/share/php5"

       ErrorLog /var/log/apache2/statistics-error.log
       TransferLog /var/log/apache2/statistics-access.log

</VirtualHost>

When everything is ready (do not forget to restart your Apache if you have enabled new modules or sites), you can try to consult the URL :
http://statistics.com/stats/gui/trunk/team/
If all seems to work, and if PHP scripts are correctly displayed in your browser, then you have finished to configure l10n-stats.

If everything works, Congratulations ! Do not hesitate to contact Nicolas Ternisien to inform him your project use l10n-stats scripts and maybe add it to the L10n-stats About page.

Customize Web Translation Statistics Style

If you would like to use a different style from KDE for your website, here is a little description of main files :

media/includes/newlayout/: This folder contains the template layouts of pages header and footer.
media/css.php: Default CSS stylesheet for KDE. The modification of these CSS will help you to customize the general web site layout (header, footer, menu,...)
media/images/: Default images for KDE website. Add your own images in this folder to customize your website.
css.inc: This stylesheets defines the statistics table style and appearance. Its modification is particularly useful if you would like to change the layout of the statistics pages.

For more information do not hesitate to consult the media/ framework documentation.

SQL Schema Description

You can download the SQL script example to see what you need to configure : datas-examples.sql. The SQL script in charge of creating tables is available here : tables-creation.sql.

Table l10n_revisions

This table contains branches (aka "revision") of your project. The root folder containing the working copy of your repository must contain each branches in one folder. In our example, these revisions are trunk and stable.

The second value allow the administrator to disabled the wanted revisions, using the following values : 1 for an enabled revision, and 0 for a disabled one.

The third value let you select the revision as main revision of your project. Only one revision could have this value at 1.

Tip : If your repository does not look like this, you can use multiple svn checkout command to create a working copy that looks like the required one. See your SVN manual for more information.

INSERT INTO `l10n_revisions` VALUES ('trunk',1,1);
INSERT INTO `l10n_revisions` VALUES ('stable',1,0);

Table l10n_teams

This table contains all teams of your localization project. The first value is the team code (and the name of the folder where PO files can be found). The second value is for displaying the name of the team.

INSERT INTO `l10n_teams` VALUES ('af','Afrikaans');
INSERT INTO `l10n_teams` VALUES ('ar','Arabic');
INSERT INTO `l10n_teams` VALUES ('az','Azerbaijani');
INSERT INTO `l10n_teams` VALUES ('be','Belarusian');

Table l10n_guicfgpackage

This table contains the different packages of your project for GUI (in fact, these packages represents the different folders of your repository tree).

This allows you to group PO files of your localization project easily

INSERT INTO `l10n_guicfgpackage` VALUES ('kdeaddons');
INSERT INTO `l10n_guicfgpackage` VALUES ('kdeadmin');
INSERT INTO `l10n_guicfgpackage` VALUES ('kdeartwork');
INSERT INTO `l10n_guicfgpackage` VALUES ('kdebase');

Table l10n_guicfgessential

This table contains files and packages which are considered as essential for your project.

It means that these files must be translated in priority by new localization teams. It refers in fact to the Essential pages of the generated web site.

The 'f' signify an essential file, and the 'p' an essential package folder. The second value is the name of the file/package, and finally, the last value is the minimal translation state required for considering localization nearly ready to use.

INSERT INTO `l10n_guicfgessential` VALUES ('f','desktop_l10n.po',75);
INSERT INTO `l10n_guicfgessential` VALUES ('f','desktop_kdelibs.po',75);
INSERT INTO `l10n_guicfgessential` VALUES ('f','kdelibs.po',90);
INSERT INTO `l10n_guicfgessential` VALUES ('p','kdebase',75);
INSERT INTO `l10n_guicfgessential` VALUES ('f','desktop_l10n.po',75);
INSERT INTO `l10n_guicfgessential` VALUES ('f','desktop_kdelibs.po',75);
INSERT INTO `l10n_guicfgessential` VALUES ('f','kdelibs.po',90);
INSERT INTO `l10n_guicfgessential` VALUES ('p','kdebase',75);

Tables for Documentation statistics

For documentation generation configuration SQL tables, use the same templates SQL as above, and simply replace the l10n_gui prefix by this one l10n_doc. The options are identical to the one described in GUI generation.

Tip : You can also, if you want, use same tables by reconfiguring tables name in your config file, too have same tables for Doc and Gui profiles.

Required tree of localization repository

As you can see in the previous paragraph, you need to represent exactly what your project looks like. This helps l10n-stats to know how to generate web site tree, but also where to find PO files in your working copy.

To sum up, here is the kind of working copy l10n-stats projects will look for :

The scripts need to find the following working copy tree :

$config["svn_local_dir"] /

l10n_revisions(rev) /

l10n/

l10n_teams(teamcode) /

$config["svn_message_folder"] /

l10n_[gui|doc]package(package)

SQL tables(field name) are marked in bold, and config entry in italic. Other things are static to every working copy.

Example for GUI PO files retrieving :

/home/stats/
  trunk/l10n/
    fr/messages
      package1
      package2
    en/messages
      package1
      package2
  stable/l10n/
    fr/messages
      package1
      package2
    en/messages
      package1
      package2

Example for Documentation PO files retrieving :

/home/stats/
  trunk/l10n/
    fr/docmessages
      package1
      package2
    en/docmessages
      package1
      package2
  stable/l10n/
    fr/docmessages
      package1
      package2
    en/docmessages
      package1
      package2

If your current SVN repository do not respect this architecture, you can easily build a similarly one using multiple svn checkout, for example, for each revisions, and language. Please however note that this architecture seems to be quite powerful because it is used by KDE, which is a very big project.

For more details, see the KDE SVN repository, on which l10n-stats is based.

Troubleshootings

My scripts could not be launched, the following error appears (or a similar one) :

ERROR Documentation statistics stats-datas.script.php ():
:
    status=error
    started=3/3/2006 23:29:0
    mesg=There's another stats-datas.script.php instance running. Aborting...

This error generally occurs when you try to launch at the same time the same script. It can also occur if the previous execution failed. In this case, you need to remove the lock file /tmp/stats-datas.script.php.lock.

Use a newer version

If you want to use a more recent version of l10n-stats, it should be really easy if you have correctly followed this documentation.

First, copy your template folder to another place and be sure the config file is still in your home (.l10n-stats.conf). Once these tasks are done, simply remove old l10n-stats folder, download and uncompress the new version, and recopy template folder to this new version. Just watch new parameters and documentation about new features and test it. That's all !

Credits

Contact Nicolas Ternisien <nicolas.ternisien@gmail.com> if you have any questions or requests about l10n-stats project. Nicolas Ternisien is the current maintainer of l10n-stats and the l10n.kde.org web site.

You can still visit the official web site of l10n-stats project : http://l10n.kde.org/about-stats.php

The original author of these scripts is Claudiu Costin.

GUI statistics

The following people made nice suggestions for l10n-stats project, about GUI statistics generation (alphabetical order) :

Dimitrios Kamenopoulos <el97146@mail.ntua.gr>: Hunted many spelling and grammatically mistakes I've made.
Dwayne Bailey <dwayne@obsidian.co.za>: Requested the history graphics to see overall progress of translation process per team and per project.
Eric Bischoff <e.bischoff@noos.fr>: HTML visual appearence suggestions, bugs reporting, found spelling mistakes.
Funda Wang <fundawang@linux.net.cn>: Spotted little bug on navigation page on second level; suggested to add usefull links in partial translation page for second level. Also observed the flawled math effect on graph bars.
Konrad Dabrowski <konraddabrowski@yahoo.co.uk>: Sent me very detailed report about math errors I've made. These errors have been very tricky because showed up on limited combination of statistics numbers.
Krzysztof Lichota <lichota@mimuw.edu.pl>: Reported a new math bug caused by little typos: fuzzy percentage in lines and totals always have been multipled by 2. Spotted a missing CVS branch variable in cvsweb URL for PO files on 3rd level statistics files.
Pedro Morais <pmmm@rnl.ist.utl.pt>: Found display bug in "Top 10", suggested the "Partial Translations" page with all features it has.
Thomas Diehl <thd@kde.org>: Lots of suggestions for GUI Statistics content, page format, structure and many other hints.

Documentation statistics

The following people made nice suggestions for l10n-stats project, about Documentation generation (alphabetical order) :

Amilcar do Carmo Lucas <amilcar@kdevelop.org>: Observed that "About" and "Help" pages are missing in KDE DOC Statistics.
Erik K. Pedersen <erik@binghamton.edu>: Reported that PO files in kdelibs wouldn't be taken in account when grabbing statistics.
Pedro Morais <pmmm@rnl.ist.utl.pt>: Pushed me ahead when I was lazy. Promptly pointed me to configure and fix stats when KDE switched to new releases.
Federico Zenith <zenith@chemeng.ntnu.no>: Sugested better explanation for "update" icon.

Last update : February, 14 2008.