Skip to content

Commit 12fcc87

Browse files
ikreymerikreymer
authored
Localization Support (#647)
* add localization utilities: - add locmanager to support extract, update, remove, list using pybabel - add po2csv/csv2po conversion with translate-utils - docs: add localization.rst to manual! * add language switch header (via header.html) to all pages if more than one locale is present. * localization: wrap more text strings in templates in existing templates * docs: - document `wb-manager i18n` commands - mention `<html lang>` setting - include csv example - add info about adding localizable text in templates * add localization to CHANGES
1 parent 0eedd15 commit 12fcc87

16 files changed

+341
-48
lines changed

‎CHANGES.rst

+11
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,17 @@ Documentation Updates:
77

88
* `New ACL header configuration <https://pywb.readthedocs.io/en/latest/manual/usage.html#config-acl-header>`_
99

10+
* `Locaalization / Multi-lingual Support Guide <https://pywb.readthedocs.io/en/latest/manual/localization.html>`_
11+
12+
13+
Localization Improvements: (`#647 <https://github.com/webrecorder/pywb/pull/647>`_)
14+
15+
* Support for extracting, updating, listing and removing localizable commands via ``wb-manager i18n`` command.
16+
17+
* UI: Add language switch header to all UI templates.
18+
19+
* Mark localizable strings in translatable in existing templates.
20+
1021

1122
Access Control Improvements:
1223

‎babel.ini

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[jinja2: pywb/templates/**.html]
2+
extensions=jinja2.ext.i18n,jinja2.ext.autoescape,jinja2.ext.with_

‎config.yaml

+3
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,6 @@ enable_memento: true
1717
# Replay content in an iframe
1818
framed_replay: true
1919

20+
locales:
21+
- en
22+
- es

‎docs/index.rst

+1
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ A subset of features provides the basic functionality of a "Wayback Machine".
1818
manual/configuring
1919
manual/access-control
2020
manual/ui-customization
21+
manual/localization
2122
manual/architecture
2223
manual/apis
2324
manual/owb-transition

‎docs/manual/localization.rst

+147
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
.. _localizaation:
2+
3+
Localization / Multi-lingual Support
4+
------------------------------------
5+
6+
pywb supports configuring different language locales and loading different language translations, and dynamically switching languages.
7+
8+
pywb can extract all text from templates and generate CSV files for translation and convert them back into a binary format used for localization/internationalization.
9+
10+
(pywb uses the `Babel library <http://babel.pocoo.org/en/latest/>`_ which extends the `standard Python i18n system <https://docs.python.org/3/library/gettext.html>`_)
11+
12+
Locales to use are configured in the ``config.yaml``.
13+
14+
The command-line ``wb-manager`` utility provides a way to manages locales for translation, including generatin extracted text, update translated text.
15+
16+
Adding a Locale and Extracting Text
17+
===================================
18+
19+
To add a new locale for translation and automatically extract all text that needs to be translated, run::
20+
21+
wb-manager i18n extract <loc>
22+
23+
The ``<loc>`` can be one or more supported two-letter locales or CLDR language codes. To list available codes, you can run ``pybabel --list-locales``.
24+
25+
Localization data is placed in the ``i18n`` directory, and translatable strings can be found in ``i18n/translations/<locale>/LC_MESSAGES/messages.csv``
26+
27+
Each CSV file looks as follows, listing source string and an empty string for the translated version::
28+
29+
"location","source","target"
30+
"pywb/templates/banner.html:6","Live on",""
31+
"pywb/templates/banner.html:8","Calendar icon",""
32+
"pywb/templates/banner.html:9 pywb/templates/query.html:45","View All Captures",""
33+
"pywb/templates/banner.html:10 pywb/templates/header.html:4","Language:",""
34+
"pywb/templates/banner.html:11","Loading...",""
35+
...
36+
37+
38+
This CSV can then be passed to translators to translate the text.
39+
40+
(The extraction parameters arae configured to load data from ``pywb/templates/*.html`` in ``babel.ini``)
41+
42+
43+
For example, the following will generate translation strings for ``es`` and ``pt`` locales::
44+
45+
wb-manager i18n extract es pt
46+
47+
48+
The translatable text can then be found in ``i18n/translations/es/LC_MESSAGES/messages.csv`` and ``i18n/translations/pt/LC_MESSAGES/messages.csv``.
49+
50+
51+
The CSV files should be updated with a translation for each string in the target column.
52+
53+
The extract commannd add any new strings without overwriting existing translations, so it is safe to run multiple times.
54+
55+
56+
Updating Locale Catalog
57+
=======================
58+
59+
Once the text has been translated, and the CSV files updated, simply run::
60+
61+
wb-manager i18n update <loc>
62+
63+
This will parse the CSVs and compile the translated string tables for use with pywb.
64+
65+
66+
Specifying locales in pywb
67+
==========================
68+
69+
To enable the locales in pywb, add one or more locales can be added to the ``locales`` key in ``config.yaml``, ex::
70+
71+
locales:
72+
- en
73+
- es
74+
75+
Single Language Default Locale
76+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
77+
78+
pywb can be configured with a default, single-language locale, by setting the ``default_locale`` property in ``config.yaml``::
79+
80+
81+
default_locale: es
82+
locales:
83+
- es
84+
85+
86+
With this configuration, pywb will automatically use the ``es`` locale for all text strings in pywb pages.
87+
88+
pywb will also set the ``<html lang="es">`` so that the browser will recognize the correct locale.
89+
90+
91+
Mutli-language Translations
92+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
93+
94+
If more than one locale is specified, pywb will automatically show a language switching UI at the top of collection and search pages, with an option
95+
for each locale listed. To include English as an option, it should also be added as a locale (and no strings translated). For example::
96+
97+
locales:
98+
- en
99+
- es
100+
- pt
101+
102+
will configure pywb to show a language switch option on all pages.
103+
104+
105+
Localized Collection Paths
106+
==========================
107+
108+
When localization is enabled, pywb supports the locale prefix for accessing each collection with a localized language:
109+
If pywb has a collection ``my-web-archive``, then:
110+
111+
* ``/my-web-archive/`` - loads UI with default language (set via ``default_locale``)
112+
* ``/en/my-web-archive/`` - loads UI with ``en`` locale
113+
* ``/es/my-web-archive/`` - loads UI with ``es`` locale
114+
* ``/pt/my-web-archive/`` - loads UI with ``pt`` locale
115+
116+
The language switch options work by changing the locale prefix for the same page.
117+
118+
Listing and Removing Locales
119+
============================
120+
121+
To list the locales that have previously been added, you can also run ``wb-manager i18n list``.
122+
123+
To disable a locale from being used in pywb, simply remove it from the ``locales`` key in ``config.yaml``
124+
125+
To remove data for a locale permanently, you can run: ``wb-manager i18n remove <loc>``. This will remove the locale directory on disk.
126+
127+
To remove all localization data, you can manually delete the ``i18n`` directory.
128+
129+
130+
UI Templates: Adding Localizable Text
131+
=====================================
132+
133+
Text that can be translated, localizable text, can be marked as such directly in the UI templates:
134+
135+
1. By wrapping the text in ``{% trans %}``/``{% endtrans %}`` tags. For example::
136+
137+
{% trans %}Collection {{ coll }} Search Page{% endtrans %}
138+
139+
2. Short-hand by calling a special ``_()`` function, which can be used in attributes or more dynamically. For example::
140+
141+
... title="{{ _('Enter a URL to search for') }}">
142+
143+
144+
These methods can be used in all UI templates and are supported by the Jinja2 templating system.
145+
146+
See :ref:`ui-customizations` for a list of all available UI templates.
147+

‎extra_requirements.txt

+1
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@ uwsgi
55
ujson
66
pysocks
77
lxml
8+
translate_toolkit

‎pywb/apps/rewriterapp.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,8 @@ def __init__(self, framed_replay=False, jinja_env=None, config=None, paths=None)
7272

7373
self.jinja_env.init_loc(self.config.get('locales_root_dir'),
7474
self.config.get('locales'),
75-
self.loc_map)
75+
self.loc_map,
76+
self.config.get('default_locale'))
7677

7778
self.redirect_to_exact = config.get('redirect_to_exact')
7879

‎pywb/manager/locmanager.py

+109
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
import os
2+
import os.path
3+
import shutil
4+
5+
from babel.messages.frontend import CommandLineInterface
6+
7+
from translate.convert.po2csv import main as po2csv
8+
from translate.convert.csv2po import main as csv2po
9+
10+
11+
ROOT_DIR = 'i18n'
12+
13+
TRANSLATIONS = os.path.join(ROOT_DIR, 'translations')
14+
15+
MESSAGES = os.path.join(ROOT_DIR, 'messages.pot')
16+
17+
# ============================================================================
18+
class LocManager:
19+
def process(self, r):
20+
if r.name == 'list':
21+
r.loc_func(self)
22+
elif r.name == 'remove':
23+
r.loc_func(self, r.locale)
24+
else:
25+
r.loc_func(self, r.locale, r.no_csv)
26+
27+
def extract_loc(self, locale, no_csv):
28+
self.extract_text()
29+
30+
for loc in locale:
31+
loc_dir = os.path.join(TRANSLATIONS, loc)
32+
if os.path.isdir(loc_dir):
33+
self.update_catalog(loc)
34+
else:
35+
os.makedirs(loc_dir)
36+
self.init_catalog(loc)
37+
38+
if not no_csv:
39+
base = os.path.join(TRANSLATIONS, loc, 'LC_MESSAGES')
40+
po = os.path.join(base, 'messages.po')
41+
csv = os.path.join(base, 'messages.csv')
42+
po2csv([po, csv])
43+
44+
def update_loc(self, locale, no_csv):
45+
for loc in locale:
46+
if not no_csv:
47+
loc_dir = os.path.join(TRANSLATIONS, loc)
48+
base = os.path.join(TRANSLATIONS, loc, 'LC_MESSAGES')
49+
po = os.path.join(base, 'messages.po')
50+
csv = os.path.join(base, 'messages.csv')
51+
52+
if os.path.isfile(csv):
53+
csv2po([csv, po])
54+
55+
self.compile_catalog()
56+
57+
def remove_loc(self, locale):
58+
for loc in locale:
59+
loc_dir = os.path.join(TRANSLATIONS, loc)
60+
if not os.path.isdir(loc_dir):
61+
print('Locale "{0}" does not exist'.format(loc))
62+
return
63+
64+
shutil.rmtree(loc_dir)
65+
print('Removed locale "{0}"'.format(loc))
66+
67+
def list_loc(self):
68+
print('Current locales:')
69+
print('\n'.join(' - ' + x for x in os.listdir(TRANSLATIONS)))
70+
print('')
71+
72+
def extract_text(self):
73+
os.makedirs(ROOT_DIR, exist_ok=True)
74+
75+
CommandLineInterface().run(['pybabel', 'extract', '-F', 'babel.ini', '-k', '_ _Q gettext ngettext', '-o', MESSAGES, './', '--omit-header'])
76+
77+
def init_catalog(self, loc):
78+
CommandLineInterface().run(['pybabel', 'init', '-l', loc, '-i', MESSAGES, '-d', TRANSLATIONS])
79+
80+
def update_catalog(self, loc):
81+
CommandLineInterface().run(['pybabel', 'update', '-l', loc, '-i', MESSAGES, '-d', TRANSLATIONS, '--previous'])
82+
83+
def compile_catalog(self):
84+
CommandLineInterface().run(['pybabel', 'compile', '-d', TRANSLATIONS])
85+
86+
87+
@classmethod
88+
def init_parser(cls, parser):
89+
"""Initializes an argument parser for acl commands
90+
91+
:param argparse.ArgumentParser parser: The parser to be initialized
92+
:rtype: None
93+
"""
94+
subparsers = parser.add_subparsers(dest='op')
95+
subparsers.required = True
96+
97+
def command(name, func):
98+
op = subparsers.add_parser(name)
99+
if name != 'list':
100+
op.add_argument('locale', nargs='+')
101+
if name != 'remove':
102+
op.add_argument('--no-csv', action='store_true')
103+
104+
op.set_defaults(loc_func=func, name=name)
105+
106+
command('extract', cls.extract_loc)
107+
command('update', cls.update_loc)
108+
command('remove', cls.remove_loc)
109+
command('list', cls.list_loc)

‎pywb/manager/manager.py

+11
Original file line numberDiff line numberDiff line change
@@ -441,6 +441,17 @@ def do_acl(r):
441441
ACLManager.init_parser(acl)
442442
acl.set_defaults(func=do_acl)
443443

444+
# LOC
445+
from pywb.manager.locmanager import LocManager
446+
def do_loc(r):
447+
loc = LocManager()
448+
loc.process(r)
449+
450+
loc_help = 'Generate strings for i18n/localization'
451+
loc = subparsers.add_parser('i18n', help=loc_help)
452+
LocManager.init_parser(loc)
453+
loc.set_defaults(func=do_loc)
454+
444455
# Parse
445456
r = parser.parse_args(args=args)
446457
r.func(r)

‎pywb/rewrite/templateview.py

+9-3
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,8 @@ def __init__(self, paths=None,
9898
assets_env.resolver = PkgResResolver()
9999
jinja_env.assets_environment = assets_env
100100

101+
self.default_locale = ''
102+
101103
def _make_loaders(self, paths, packages):
102104
"""Initialize the template loaders based on the supplied paths and packages.
103105
@@ -117,16 +119,19 @@ def _make_loaders(self, paths, packages):
117119

118120
return loaders
119121

120-
def init_loc(self, locales_root_dir, locales, loc_map):
122+
def init_loc(self, locales_root_dir, locales, loc_map, default_locale):
121123
locales = locales or []
124+
locales_root_dir = locales_root_dir or os.path.join('i18n', 'translations')
125+
default_locale = default_locale or 'en'
126+
self.default_locale = default_locale
122127

123128
if locales_root_dir:
124129
for loc in locales:
125-
loc_map[loc] = Translations.load(locales_root_dir, [loc, 'en'])
130+
loc_map[loc] = Translations.load(locales_root_dir, [loc, default_locale])
126131
#jinja_env.jinja_env.install_gettext_translations(translations)
127132

128133
def get_translate(context):
129-
loc = context.get('env', {}).get('pywb_lang')
134+
loc = context.get('env', {}).get('pywb_lang', default_locale)
130135
return loc_map.get(loc)
131136

132137
def override_func(jinja_env, name):
@@ -160,6 +165,7 @@ def quote_gettext(context, text):
160165

161166
self.jinja_env.globals['locales'] = list(loc_map.keys())
162167
self.jinja_env.globals['_Q'] = quote_gettext
168+
self.jinja_env.globals['default_locale'] = default_locale
163169

164170
@contextfunction
165171
def switch_locale(context, locale):

‎pywb/static/default_banner.js

+2-2
Original file line numberDiff line numberDiff line change
@@ -182,7 +182,7 @@ This file is part of pywb, https://github.com/webrecorder/pywb
182182
ancillaryLinks.appendChild(calendarLink);
183183
this.calendarLink = calendarLink;
184184

185-
if (typeof window.banner_info.locales !== "undefined" && window.banner_info.locales.length) {
185+
if (typeof window.banner_info.locales !== "undefined" && window.banner_info.locales.length > 1) {
186186
var locales = window.banner_info.locales;
187187
var languages = document.createElement("div");
188188

@@ -317,4 +317,4 @@ This file is part of pywb, https://github.com/webrecorder/pywb
317317
}
318318
}
319319

320-
})();
320+
})();

‎pywb/templates/base.html

+1
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
<!-- jquery and bootstrap dependencies query view -->
1010
<link rel="stylesheet" href="{{ static_prefix }}/css/bootstrap.min.css"/>
1111
<link rel="stylesheet" href="{{ static_prefix }}/css/font-awesome.min.css">
12+
<link rel="stylesheet" href="{{ static_prefix }}/css/base.css">
1213

1314
<script src="{{ static_prefix }}/js/jquery-latest.min.js"></script>
1415
<script src="{{ static_prefix }}/js/bootstrap.min.js"></script>

0 commit comments

Comments
 (0)