Open Atrium: Solving the Translation Puzzle
We launched translate.openatrium.com earlier this week to support translating Open Atrium into more than a dozen languages. It currently ships out of the box in English, Spanish, and Arabic, but we want to grow this. To do this we built a localization server that provides downloadable, up-to-date translations that are automatically repackaged every few hours and that allows people to post their own translations and fixes to the server while they’re doing them on their own sites.

The tools that make this possible have been in the works for some time now. The localization client allows for on page translations of UI strings, and the localization server lets you keep a centralized translations repository for all modules, versions, and languages. We’ve had the tools for awhile, however, in order to realize just how hard it is to actually do and maintain translations in Drupal, you need to try it.
In the case of Open Atrium, we’re not talking about a Drupal module or a single site. Open Atrium is a full featured Drupal intranet distribution that uses Drupal core, some contributed modules, and some custom specific ones, and all of this this is deployed to many sites. This is one of those times when it feels like you have all the pieces to do what you want to do, but it’s still hard to figure out how to make everything work together in a human friendly way to get the job done.
So for Open Atrium, we want to:
- Provide a downloadable package for a simple installation in any of the supported languages.
- Set up some channel to allow people to easily contribute and improve translations.
- Keep everything up-to-date with our release schedule and the quickly moving codebase.
- Allow Open Atrium users to easily keep their translations up to date with the latest version.
Nope, it’s just not that easy.
With a contributor-based translation system like this, the code updates come before the translations. So we first release the modules, then wait for the translations to come in, and finally release the translations. There’s no way we can deliver updated translations with each release unless we have a full team dedicated to translate pre-release versions.
There are about 50 modules in Open Atrium besides Drupal core, and that makes updating single module translations unworkable. To get around this issue, we came up with a meta-package idea – to bundle the code into different packages that are imported by the extractor, creating some fictitious projects (and versions) that can be either translated, handled, or released separately. This gave us an atrium_features package, an atrium_themes, an atrium_custom, and a big atrium package that includes all the others.

Now for how to keep the translations up-to-date. There is a very little known feature built into the localization client and server that allows a client to automatically post translations over xml-rpc, using some shared key system that provides client authentication to the server. This is important.
While the localization server allows for the downloading of translations, they’re repackaged every time and therefore not really suitable for public downloads (especially if you expect a lot of them). To get around this we’ve built another custom module that automatically repackages all translations periodically and makes them available for downloading.

We’re excited to test this out and keep working on it, as some kind of l10n server will likely be in the future of Drupal to handle core and module translations. Hopefully, some of the work we’re doing can be later reused for that.
Coming next – automatically updating your site’s translations (which is why we really needed the packaging module).
And for fun, here’s a peak at the full design for translate.openatrium.com created by Saman that I just love.

Fantastic
Really fantastic work all around, keep up the great work. I hope that my team will be contributing some translation stuff for OpenAtrium stuff this year …
Great to see focus on the translation server
Just some quick questions: How are you going to organize the validation of translation suggestions? In a previous attempt to use the great potential of i10n server the biggest problem was not to collect suggestions but to validate them and having someone taking care only quality translations are getting in.
If you are working on a packaging script – would it be a huge step to provide a complete download of Open Atrium + language X including the current atrium and Drupal core translation? Such a feature would be a great improvement for drupal.org since installing Drupal in German is more complex than it could be.
About validation, that
About validation, that certainly needs to be done manually so there will be at least one reviewer per language. Also, if you’ve tried using the l10n server for that before, it is quite hard to handle it for many strings so we have some tools in the works to help reviewers do their work like having a mass approval page or trusted translators that can skip validation.
About packaging translations with the distribution, we are aiming at including some minimum amount of strings on the package (the very minimum to get started with the install) and then automatically download the strings for the chosen language.
Jose
Translation servers
Translation servers should be kept per-language.
As long as we do not solve this issue in a larger scope.
Otherwise, we have different, possible sets of possibly valid translations of a certain language. That is strictly against collaboration (because one translation server needs to win) and hinders evolution of translations as well as translation teams.
Please revert this.
Aside from that, keep up this awesome stuff! I really adore you guys! :)
Though I agree we should wrap
Though I agree we should wrap our solutions and our minds around the ‘collaboration’ idea, I’m not that sure we need to nor want to have that ‘unique valid translations’.
When building a very targetted Drupal solution like Open Atrium is for intranets, you may want to use different translations for some strings so this is one reason why the ‘only one official translation’ is not really an option. The other reason is that we don’t have yet the infrastructure set up for this universal sharing….
So please, follow on the comments below (Gabor’s) about how we can actually build that infrastructure… But really, we cannot use something we don’t have yet…
Jose Reyero
I tend to agree that since
I tend to agree that since Atrium is a whole solution by itself, it is better to be able to have an Atrium specific translation. It also makes it easier on the translator to pick an expression that is “valid” in the context of Atrium without having to consider if this would still be valid in a more general context.
It is also possible to give and “Atrium tone” to the translation which can also improve the user experience IMO.
Of course, if it is also possible to share back some of the work done in Atrium back to a more generalized (language specific) translation server (for contrib modules and all) that would be even better.
Channelling back and forth
Well, your use of the localization server and other possible distributions raise an interesting question. The Open Atrium distribution obviously includes contrib modules from Drupal.org and custom modules / features only available from your repository, or at least not available from drupal.org. Like Linux distributions, translations for downstream distributions would be good to share upstream, so the translations on drupal.org (or in some cases the respective translation team servers) can be kept up to date. So contributors to Open Atrium flow back to the community even for people, who are not using this distribution. Did you think about possible solutions for this?
I was thinking about possibly supporting “downstream” localization servers on a localization server, so for example, a translation team can still keep using their local environment with their forums/tools set up, while drupal.org could become an “aggregator of translations” of code hosted there, and would aggregate translations for components from distributions like Open Atrium. I am not sure how should we keep attribution of translations in this case (hall of fames are powerful as seen from the issue queue of Localization server).
Would be interested in your thoughts on that.
Hi Gabor. Yes, we share yor
Hi Gabor. Yes, we share yor concerns about how we can best contribute back all the stuff in the server to drupal.org. For now we’ve just posted some po files in the few cases we had some full translation for a contrib module…
However, as you say this is not that straight forward and we certainly need some better way to handle translations across servers/repositories and also credit for translators. So at this stage we’ll be just developing some tools and contribute back the translations when possible. I hope that some of these tools will be useful later as part of a bigger workflow.
About how it should work I basically share your idea of having a big central server aggregating all translations, then being able to merge translations upstream and downstream. Our packaging module may be a first step for this as it allows easy packaging of files available on (almost) permanent urls.
As for the attribution issue, I think one option to handle it accross servers would be to decouple translations from system users and having a full url for attribution (that can be either a server or an user in a server). Or maybe having a ‘drupal.org’ user id as global id for translators. A drupal.org OpenID will qualify for this, and it will allow to link translations to drupal.org users.
Anyway, maybe we should get started with drupal.org l10n server asap so we can start building on it. Lets start community translations first, then we can think about how to package and deploy them (?)
Or if you think this is an step worth giving it some more time and thinking, well, we’ll be sharing our experience and the tools we are building, maybe that can be some help to make more informed decissions for the ‘big drupal.org translation server’.
We can also talk at Drupalcon Paris. I hope for that date we’ll have already worked out some full set up and translation workflow.
Jose
on the central server
I am on doing the central server, since Gerhard gave me some gentle nudges (again): http://twitter.com/gaborhojtsy/status/2740776098
Great initiative
Great work Jose! As usual, great ideas are not necessarily easy to implement… good thing they have you to do it — keep it up!
Demo Portal
As one of the funders of drupal.org.il and one of it’s first translators to Hebrew, I really appriciate your work.
The video is great – but it downloads slowly. Looking forwards to a live demo portal.
Amnon Levav http://levavie.com
Beautiful work
Great work, as always.