PulkoMandy's blog

Locale Kit: the big picture

Blog post by PulkoMandy on Sat, 2009-06-27 19:14

almost-UML diagram of the locale kitalmost-UML diagram of the locale kit

This week, I finally got the plaintext catalog add on working. Then today, Oliver reviewed my work and we had a meeting on IRC. We agreed on some changes to the internal architecture of the locale kit, and also to the classes I added. Some classes in this kit have unappropriate names, and the kit was designed with zeta compatibility in mind, whereas in today's Be-World it seems more important to focus on gettext. I summarized our meeting in this somewhat-UML diagram. It's not really an UML object diagram as I added informations about the workflow and the tools to use when working on a localized application, and a lot of notes and remarks. I think it shows well the status of the kit: a clearly designed API, but some missing code and a lot of TODO notes. I did not fully reported the parameters of the methods and the attributes of all the classes, look at the headers if you need some information on that. This diagram is not meant as a technical reference, but as a colourful and efficient way of cleaning up my personal notes. I think I'm better at organizing my tought in a spacial way than laying them down linearly in a text. Click on the image to enjoy it at full size :)

Locale Kit: (somewhat) weekly report

Blog post by PulkoMandy on Fri, 2009-06-19 18:42

Mid-term evaluations for GSoC are already coming...

I'm still working on the catalogs for translating applications. I got the system working and integrated it into Haiku, so now any application can be translated. However, there is still a lot of work to do. I'm now working on a plaintext catalog add on.

Catalogs are the files that store translated strings. There is a catalog add-on called "default" that is used in applications. The problem is that the files for this add on are stored in binary form (as archived BMessages), so it's not easy to translate them.

So, the plaintext catalog add-on will allow developers to extract strings from their source programs and easily translate them in a plain text file. Then it will be possible to convert it to a binary file for use in the application. This is simpler to code than a full featured translation tool (that may come later), and also allow testing of the add-on system.

This is about everything for the catalog system, the other part of the work on this project is the formatting toolkit with functions for formatting dates, times, and numbers. This should be done using ICU, a library that provides all the needed classes, but will be wrapped in the already designed Locale Kit API, which feels more in accordance with the rest of the BeAPI. ICU will also be used to manage a big part of the preflet : language listing, and mapping languages to presets for all the settings.

The last part of the project will be mostly centered on the preflet. It will have to send a message to all the application when the user select a different language, so all the opened window are refreshed automagically.

I have a little problem with my debian setup this week, and for now I can't build haiku under debian anymore. But I will try to build directly under haiku if possible to keep working.
Also, I just finished school exams last week, so from now I will have a lot more free time and I hope the project will speed up a lot.

Locale Kit: everything about catalogs and catalog add ons

Blog post by PulkoMandy on Mon, 2009-06-01 19:52

Today I have started to write a catalog add-on to save catalogs in plain text for easy translation. I've spent some time looking at the involved C++ classes, and here is what I found.

A catalog is a collection of strings, stored as pairs. It is used in the locale kit to translate the text in an application to the system language at runtime. When an application starts, it asks the locale roster to find its catalog and return it back. Then, each time a strig needs to be displayed, it goes trough the catalog and is translated automatically.

Catalogs are stored on disk, and can use various formats. For example, there are files originating from zeta's locale kit, unixish po files, and our own format or haiku. The app doesn't need to know which of them it is using, so finding the right one is the job of the locale roster. All the different formats are handled by catalog add-ons. Every catalog knows which add on it needs to use to do all the work of loading and saving. An EditableCatalog can be modified, it's not meant to be used by a regular application, but only by the apps made for editing catalogs. For now, we will use collectcatkeys to extract all the strings from a sourcecode and dump them to a plain text catalog. Then, linkcatkeys will convert this catalog to one in the native format, which is faster to parse at runtime. Later it will be possible to create an app that allows direct editing of the compiled catalog with a nice graphical interface.

This will allow me to easily create test catalogs in different languages and use them in my HelloWorld test app. Then, i will add language swtching to the locale preflet, and test it so the testapp updates all its strings in realtime when a new language is selected in the preflet.

Locale Kit : status update n.2

Blog post by PulkoMandy on Wed, 2009-05-27 11:51

These two weeks I've been quite busy with other things, so the project didn't move as much as I wanted. However, I managed to get the catalog engine to internationalize an app for the first time. It's not a big application, just a very simple Hello World test program. And the lack of a tool for translating catalogs means I had to edit them by hand to get the translation done.
I will be working in a Catalog AddOn writing a catalog as full text for easier editing.

So, to test the program, you have to

1) run make
2) mkdir obj.x86/locale
3) mkdir obj.x86/locale/catalogs
4) mkdir obj.x86/locale/x-vnd.Be-HelloWorldSample
5) cpp HelloWindow.cpp > HelloWindow.precpp
6) collectcatkeys -s HELO HelloWindow.precpp -o obj.x86/locale/catalogs/x-vnd.Be-HelloWorldSample/english.catalog
7) run the application
8) you can edit the last occurence of "Hello World!" in the catalog file to change what the app displays.

It seems that attaching catalog as resource or attribute to the executable file doesn't work yet.
You can get the project here: http://dl.free.fr/vqHTlU63B

Feedback is welcome :)

Locale Kit : status update n.1

Blog post by PulkoMandy on Sun, 2009-05-03 16:34

The work on the Locale Kit as part of the Summer of Code has already started :)
This week we have been working on proper integration of my work in the Haiku tree. So you can now checkout Haiku from svn and get the Locale Kit as part of it.
Of course, some parts are still broken (or not yet written), but some of the tests seems to be already working.

I'm now looking at ICU sourcecode to study the best way to integrate it in Haiku.

So, nothing much to say this week, but a lot of work done.

GSoC project : Internationalization for Haiku

Blog post by PulkoMandy on Tue, 2009-04-21 08:02

Hello world !
As you know, I am one of the selected students for this year summer of code. In this post I will introduce myself and give some details about my project.

My name is Adrien Destugues, some of you may know me as PulkoMandy as i've been lurking on the irc channel and mailing lists for some time. I already applied for the Summer of Code and Haiku Code Drive last year but unfortunately I was not selected. This year it went better :)
I'm studying electronics and computer science at the ENSSAT (École Nationale Supérieure de Sciences Appliquées et de Technologie), in Lannion, France. I used to run BeOS as my main operating system for some time, but I now switched to Linux. I have a running Haiku install on my hard disk, but as my network card is not supported, I don't use it much for everyday work. I hope this will change soon.

Now, some information on my project: the idea was born because we were talking about "which os is better" on the ENSSAT mailing list, between linux, mac os X and windows (the kind of endless discussion student have all the day long ;)). I sent a link to Haiku homepage to point the fact that there was other alternatives, and someone replied "hey, it doesn't even speaks french!". This fact is quite annoying for an operating system targeted to regular users, and not powerful computer geeks. Not everyone speaks english, and this is what an user will see first when running Haiku for the first time. So I decided to improve that and get Haiku internationalized. I did some research on the web and found informations about localization under BeOS. There was a locale kit in zeta, but the Haiku team had already started another project as part of Open Tracker. It was not integrated into Haiku because of some problems, the main one being the way the Be API is used to create windows. As the translated texts have a different length than the original ones, the window may look bad with overlapping texts and other problems similar to font sensivity.
The Layout system can now solve that, so it's about time to get the Locale Kit back to work.

The locale kit must do various things. It provides a way to translate strings, of course, but also offers services for formatting date, time, numbers and currencies. Each country have different rules for that, and any program can ask the locale kit to do the formatting properly. Another service is a natural order sorter, which sort a set of strings alphabetically but with special rules for accents, diacritics, and so on. Of course, all of this has to be done at runtime. This way the language is selected and configured with a simple preflet and all opened applications are instantly translated.
Some kind of compatibility with zeta locale kit and gnu gettext would be nice, it would allow to use translations from other systems, either directly or by using a conversion tool.

The basic layout for all that is already in place in OpenTracker's Locale Kit. However, most of the code is not written, for example there is no code for date formatting, for loading Zeta's catalog files. The API is well designed, so I will keep most of it, and fill in the missing functions. The ICU library will be used as a backend for everything except the translation. It provides all the needed data for doing all the formatting work properly.

The translation can be done with string-based or key-based lookup. The Locale Kit handles a list of catalogs, some of them are specific to an application and some are system- or user-wide. Catalogs can be stored on hard disk as plain files, resources in an application executable or bfs attributes. Catalog format can be added in the form of add-ons.

List of project goals
1) Integrate OpenTracker's Locale Kit to the Haiku build system. I've already got it building as part of the 3rdparty/ folder, and got the basic tests working. However a proper integration will need to get out of 3rdparty and fully integrated to the build system.
2) Create an example localizable "Hello World" application and test it. It will allow two things: testing of the kit and showing to programmers the way to use this kit properly.
3) Integrate the ICU library to do the collation (alphabetical sorting) and date/time/number/currency formatting. Don't try to reproduce their huge work in getting a database for all the countries. Modify their code so it compile cleanly under gcc2 (if needed) and integrates cleanly in the Be API.
4) Design and create a preflet for easily changing the language while running Haiku. It will have the following features :
* edit the settings for date, numbers and currencies and save them as user defined settings,
* allow to change the default settings for a country for all these settings,
* allow the user to select his preferred languages in a sorted list.
5) Create tools for translating an application. We need to extract translatable strings from a .cpp source file, and to save a compiled catalog for use at runtime. The current tools are working only under Haiku, and we need them under linux as part of Haiku's build process. If needed, change the catalog format to make it easy to work with, or add an importer add-on to read and write text files.
6) Write an importer (catalog add-on) for Zeta's catalogs and test it with some applications providing a Zeta catalog.
7) Write an importer for GNU gettext format.
8) Modify the code of all the preflets and applications in haiku so they use the localization system (if there is some time left at the end of summer ;))

Syndicate content