Multi-lingual support (translated fields)

[molomby]

For supporting multilingual sites and apps, it will help if Keystone itself has some concept of multiple languages. The conversation around this so far is based on a few ideas, namely:

• The languages supported by a single app will be consistent across the app. That is, different lists or area of the app or site, will support the same set of languages.
• That in a typical multilingual system, only a subset of fields (those that are user-facing) require translation. Internal labels, statuses, descriptions, etc. and user entered data (names, posts, etc.) do not.
• If a user/API-consumer wishes to operated within a specific language, the system should automatically (by default) supply them the correct version of any multilingual fields they request.

This starts to take the shape of a technical design in which:

• A developer can configure a set of languages (and a default) at the Keystone level.
• Within a list, fields (which support it? eg. text, HTML, Markdown?) can be nominate as "multilingual". This causes:
  • Multiple columns/fields to be created for them at the DB layer
  • The GraphQL layer will continue to return a single field value but this can be controlled by field argument (or a header or cookie?)
  • In the Admin UI, multilingual fields present tabs (or similar) so their different translations can be edited/populated
  • Maybe some additional filters are added? (Eg. give me a list of all items where the blahBody field has no German translation.)

This feature likely interacts with:

• Full-text search (Full-text list search (aka. "haystacks") #320) and filtering (Full-text list filters (aka. "simple search") #319), both in terms of limiting the translation to search and using indexing appropriate for different languages. Mongo only supports a single text index per collection and, as far as I can tell, only a single language per document 🙁. This has significant storage implications if we need to support multilingual content and full-text search concurrently (which seems likely).
• Default formatting of fields (eg. dates and times). It's part of a larger i18n and l10n discussion.
• Item versioning (Adding a "change log" for lists that records the version history of items #321)
• Schema building and migrations (Knex (Postgres) schema building #298). Eg. altering the list of supported languages requires schema changes/possible data migration.

We should examine how these ideas are managed in other, related systems like Contentful.

[molomby]

Related conversations in KS 4:

• Multiple language support
• Internationalization

[troycura]

Hi,

Do we have an open server that supports now for multiple language?

Thank you.

[camcnz]

Here is how I did it on a recent KS 4 project with multiple language and field text search.
I had translatable wysiwyg fields in there too.

locales.js

exports = module.exports = {
	languageListObj: {
		en: { enLabel: 'English', naLabel: 'English' },
		fr: { enLabel: 'French', naLabel: 'Français' },
		de: { enLabel: 'German', naLabel: 'Deutsch' },
		it: { enLabel: 'Italian', naLabel: 'Italiano' },
		es: { enLabel: 'Spanish', naLabel: 'Español' },
		ja: { enLabel: 'Japanese', naLabel: '日本語' },
		zh: { enLabel: 'Chinese', naLabel: '简体中文' },
	},
};

Helper_StandardContent.js

const _ = require('lodash');
const keystone = require('keystone');
const Types = keystone.Field.Types;
const { languageListObj } = require('../locales');

let translation = {};
let name = {};

for (const key in languageListObj) {
	if (languageListObj.hasOwnProperty(key)) {
		const element = languageListObj[key];
		let fieldDepends = `translation.${key}`;
		if (key !== 'en') {
			translation[key] = { type: Types.Boolean, label: `${element.enLabel}` };
		}
		if (key !== 'en') {
			name[key] = {
				type: String,
				label: `${element.enLabel} name`,
				dependsOn: { [fieldDepends]: true },
				index: true,
				validate: {
					validator: function (v) {
						return (_.get(this, fieldDepends)) ? (v && v.length > 0) : true;
					},
					message: `Please provide a name!`,
				},
			};
} else {
			name[key] = {
				type: Types.Text,
				label: `Default name`,
				initial: true,
				index: true,
				validate: {
					validator: function (v) {
						return (v && v.length > 0);
					},
					message: `Please provide a name!`,
				},
			};
exports = module.exports = {
	translation,
	name
};

ShopProduct.js

const { languageListObj } = require('../locales');
const { translation, name } = require('./Helper_StandardContent');

const ShopProduct = new keystone.List('ShopProduct', {
	label: 'Products',
	map: { name: 'name.en' },
	autokey: { path: 'slug', from: 'name.en', unique: true },
	track: { createdAt: true, updatedAt: true },
});

ShopProduct.add(
	'Translation',
	{
		translation,
	},
	{
		name,
                brand: { type: Types.Relationship, ref: 'ShopProductBrand', createInline: true, index: true },
		brandTitle: { type: Types.Text, hidden: true },
})


ShopProduct.schema.pre('save', async function (next) {
	let item = await new Promise((resolve, reject) => {
		keystone.list('ShopProductBrand').model.findOne({ _id: this.brand }, (err, data) => {
			if (err) reject(err);
			else resolve(data);
		}).lean();
	});
	if (item) {
		this.brandTitle = item.brandTitle;
	}
	return next();
});


ShopProduct.schema.plugin(mongoosePaginate);

ShopProduct.defaultColumns = 'name.en, brand, categories, gender, homepageHero, state, publishedDate';
ShopProduct.register();

// ShopProduct.model.collection.dropIndexes();

// Create text search index.
ShopProduct.model.collection.createIndex({ 'brandTitle': 'text', 'name.en': 'text', 'name.de': 'text', 'name.fr': 'text', 'name.it': 'text', 'name.es': 'text', 'name.ja': 'text', 'name.zh': 'text' }, {
	name: "TextIndex"
	}, function (error, res) {
	if (error) {
		return console.error('failed ensureIndex with error', error);
	}
	console.log('ensureIndex succeeded with response', res);
});

Product.js

exports.getAllProducts = async (req, res) => {
	try {
		let query = (req.user && req.user.isAdmin) ? {} : { state: "published", publishedDate: { "$lte": new Date(Date.now()) } };

		const locale = req.query.locale || 'en';

		if (req.query.text) {
			query.$text = { $search: req.query.text, $language: locale };
		}

		if (req.query.gender) {
			query.gender = req.query.gender;
		}

		if (req.query.categories) {
			query.categories = req.query.categories;
		}

		let options = {
			page: req.query.page || 1,
			limit: 20,
		};

		if (req.query.text) {
			options.sort = {
				score: {
					$meta: 'textScore',
				},
			};
			options.select = {
				score: {
					$meta: 'textScore',
				},
			};
		} else {
			options.sort = {
				publishedDate: -1,
			};
		}

		const items = await ShopProduct.model.paginate(query, options);

		res.apiResponse({
			items,
		});
	} catch (err) {
		res.apiError('database error', err);
	}
};

[troycura]

Do we have an open server that supports now for multiple language?

[stale[bot]]

It looks like you haven't had a response in over 3 months. Sorry about that! We've flagged this issue for special attention. It wil be manually reviewed by maintainers, not automatically closed. If you have any additional information please leave us a comment. It really helps! Thank you for you contributions. :)

[GasimGasimzada]

I have been researching this idea for some time now and I have a proposal. Firstly, in terms of releasing, I think it will be easier to do this in two steps: Localization for the Core/API and Localization for the Admin.

Now, let's get into it.

Localization Core

Initially, we need a way to define localization for "static messages." Things like validation messages, field labels, etc. So, I suggest adding a locales parameter in Keystone instance:

const keystone = new Keystone({
  name: 'New Project',
  adapter: new MongooseAdapter(),
  locales: [localeEnUs(), localeEnUk(), localeDeutsch(), ...]
});

Locale functions

As you can see, these locales do not store string value. They are functions with specific values (I have not thought of the implementation details but just the design):

// I am using momentjs formatting here but it really doesn't matter

const localeEnUs = custom => ({
  locale: 'en-us',
  languageCode: 'en',
  languageInNative: 'English',
  defaultLongDateFormat: 'dddd, MMM Do YYYY',
  defaultShortDateFormat: 'MM/DD/YYYY',
  messages: {
      api: {
        'inbox.available.one': 'You have 1 item in your inbox',
        'inbox.available.many': 'You have %d items in your inbox',
      },
      admin: {
          ...
      },
      ...custom
   }
});

This way, all the language information is contained in a single object. If a translation does not exist in a specific language, default translation will be used.

Installing Locales via NPM

Secondly, this type of containment with objects has an advantage. Since, a lot of messages are based on internal values (validation errors, admin translations), these messages should be provided by Keystone. Since, loading all locales can increase the bundle size significantly, we can separate them into separate NPM packages:

yarn add @keystonejs/locale-en-us @keystonejs/locale-de-de

# or

yarn add @keystonejs-locales/en-us @keystonejs-locales/de-de

Then store these locales in a separate monorepo (let's call it keystonejs/locales). Then, people can fork and create PRs for their own localization files and push PRs to have their localization files added to the main repo.

Custom translation messages

What if we need custom translation messages that is available to our frontend app or to a custom controller? We can just pass a JS object to the locale function to have custom variables available. Example:

const customDeDe = require('./locales/de-de.json');
const customEnUk = require('@my-own-project/localizations');
const customEnUs = { ... };

const keystone = new Keystone({
  name: 'New Project',
  adapter: new MongooseAdapter(),
  locales: [localeEnUs(customEnUs), localeEnUk(customEnUk), localeDeutsch(customDeDe), ...]
});

This allows for developer do whatever they want with their locales: they can store them in the same file as Keystone instance, store them in a separate module or just a JSON file, or publish their own localization module and import the object. Sky is the limit as long as an object is passed.

Using translation strings

In keystone, we provide a simple "translate" function. Typically, these functions have a very short name. For example, we can call it t(domain, name, default, params). Here is how to use it:

const myNumber = getNumber();
if (myNumber === 1) {
  const msg = keystone.t('api', 'inbox.available.one', 'You have 1 item in your inbox'); 
} else if (myNumber > 1) {
  const msg = keystone.t('api', 'inbox.available.many', 'You have %d items in your inbox', [myNumber]); 
}

Default value is implemented during function call. This way, if there is no value in the messages object for the current language, default one will be used. If you want to print custom values, you just need to use the domain name and the key of the custom message:

const msg = keystone.t('myCustomModule', 'hello.world', 'Hello World);

Migration

We need to use keystone.t in all our validation messages instead of normal messages.

API

Now, there needs to be a way for apps to request for two things based on locale: Localized messages and localized field values (we will discuss this one later).

Language Detector

There are multiple ways to detect languages in APIs: Cookies, query strings, Accept-Language header. My understanding is that, keystone uses express server. We can use a middleware that detects the language (or write our own) and then match the found locale with accepted locales that is based on locales parameter. However, if no locale is sent to the user, we might need a fallback / default locale. We can pass a fallbackLocale / defaultLocale parameter to Keystone instance:

const keystone = new Keystone({
  name: 'New Project',
  adapter: new MongooseAdapter(),
  locales: [localeEnUs(), localeEnUk(), localeDe(), ...],
  defaultLocale: 'de-de' // default is always the first item in the `locales` array
});

Setting language

Once the language is detected, we can store it in request context; thus, use the active language withing keystone.t or custom fields to identify what to show.

Getting language info (if needed)

We can also add a field in GraphQL to retrieve locale info and messages:

{
  localeInfo {
    nativeName
    code
    locale
    messages {
       admin {
         key
         value
       }
    }
  }
}

---

Admin Localization

Admin localization consists of couple of parts: Fields, View localization, and Language Switcher.

Fields

To make fields translatable, I suggest adding a Localization field that accepts a field parameter:

const { Text, Localized } = require('@keystonejs/fields');
keystone.createList('Post', {
  fields: {
    title: { type: Localized, field: Text },
  },
});

This field uses implementation of an existing field and adds localized functionality to it. This functionality consists of storing field values in a localized manner, getting localized and all versions of the field (needed for admin), and showing localized field in admin.

Database

For Mongoose adapter, I suggest storing field in the following way:

{
  "field": {
    "en-us": "...",
    "en-uk": "..."
  }
}

For SQL adapter, I suggesting storing the field in a related table:

// Sorry, not using the SQL syntax here. Just a simple demonstration of the database

Table: posts
 - id
 - stuff here...

Table: post_localized_fields
 - post_id: FK -> posts
 - locale
 - field
 - value
UNIQUE(post_id, locale, field)

Retrieving all localized fields from GQL

We need a way to get these values from GQL. When dealing with GQL, we can add an additional field that is only available for admins (through Access Control). Example:

{
   posts {
      titleLocalized {
        locale
        value
      }
   }
}

Viewing Localized Fields

I think the easiest way to manage views for a localized field is to add Tabs on top of the field. Each tab shows the locale code the the fields are shown within the tab. When preparing the request for the view, a schema like this will be created:

{
   "en-us": "...",
   "de-de": "..."
}

However, view code has one minor issue. Labels for each input field is always visible. I suggest hiding them (screen reader friendly) when they localized and change the labels by adding localization in parenthesis. Example (not using arch-ui, just basic, static JSX to show what I mean):

<h4 className="label">Title</h4> {/* Visible Label */}
<Tab name="English">
  <label className="sr-only" htmlFor="title-en-us">Title (English)</label
  <input type="text" id="title-en-us" ... />
</Tab>
<Tab name="Deutsche">
  <label className="sr-only" htmlFor="title-de-de">Title (Deutsche)</label>
  <input type="text" id="title-de-de" ... />
</Tab>

View Localization

We might need a way to localize, not just custom fields but the entire template (all the texts etc). I have an idea to do this in an optimized fashion. Firstly, we add a field to GQL that shows a "hash" of the all localizations. This value is generated only and stored once -- when the server is initialized and in keystone instance. Let's call this field a "localizationHash."

This field can be used by all static apps. When admin app is opened, "localizationHash" is requested from the API. Then, this hash is compared with a localizationHash key in localStorage. If values are equal, it means that we have the latest localization messages. If they are not equal (update or if value does not exist), admin app requests all the messages for the current locale and stores the values in localStorage with the new localizationHash. Then, these values will be displayed immediately. This will work great because content editors will not be changing their locales all the time. They will use the locale that they prefer (e.g French content creator might like the entire view to be in French) while also edit values from other locales since these values are custom fields that are available in all locales.

---

That's All

I know it is a long post but I wanted to suggest my proposal in detail here. If you have any questions on this or don't like something, let me know. Once a specific design is agreed upon, I am willing to implement this functionality step-by-step.

Future and Other Consideration

Once there is a localization system in place, we can use Access Controls for localized content. Developer should be able to restrict certain locales of certain fields. For example, developers should be possible for restrict a French content creator to only view and update fields that are for French localization.

[anjoalre]

Any development on this!?

[GasimGasimzada]

I haven’t done anything on this because I wrote this as a proposal to be discussed. This is a big change to the framework; so, I wrote to this to get to an agreement before going into implementation.

[justintemps]

Enthusiastically support @GasimGasimzada's idea as I'm going to have to bootstrap a multilingual support for my next project and I don't think my solution will be as good.

[lucatk]

Any update on the proposal status? @GasimGasimzada
I'd be willing to contribute on this, as localization functionality is a major requirement for a project of mine intended to be migrated to KeystoneJS currently.

[justintemps]

I'd be happy to help out as well.

[stale[bot]]

It looks like there hasn't been any activity here in over 6 months. Sorry about that! We've flagged this issue for special attention. It wil be manually reviewed by maintainers, not automatically closed. If you have any additional information please leave us a comment. It really helps! Thank you for you contribution. :)

[phoniks]

I could use this on a project I'm working on, it sounds like there's some enthusiasm for the proposal. @GasimGasimzada - it's been a while since you originally proposed this, would you still be willing to lead? I would also be willing to contribute where I can. I think this is a pretty worthwhile addition and I'll wind up doing something less complete for the project I'm currently working on if this doesn't happen, so I'd much rather contribute the hours that I'm going to spend on the problem to the implementation of this proposal as well. Any comments from the maintainers? Any reason why it's a bad idea for us to start working on this?

[stale[bot]]

It looks like there hasn't been any activity here in over 6 months. Sorry about that! We've flagged this issue for special attention. It wil be manually reviewed by maintainers, not automatically closed. If you have any additional information please leave us a comment. It really helps! Thank you for you contribution. :)

[fly-oinotna]

Is there any update on this?
I'm looking for a CMS to replace Wordpress for good and KeystoneJS looked perfect for doing that, until I ran into translations: as I'm from Austria it is necessary for me to deliver a backend in german and a frontend with different languages.

Are you guys planning to do something about this or will KeystoneJS remain a one language system?

Because in that case I'll need to (sadly) look somewhere else :(

[vonba]

Just to bump this and agree – it is only now dawning on me that Keystone seems to be lacking a very basic feature that IMO will sadly make it a no starter for many people. Hoping I have missed something and that there is a workaround!

[JedWatson]

This is definitely on our roadmap, and we're planning to come out with something in the second half of this year (2021)

To put that in context, we're in the middle of working through some core updates to our GraphQL system, which will unblock a lot of higher level features including translation support, and other oft-requested features like singleton list types and orderable items in lists. The GraphQL system updates are expected to land in about a month (late June).

A few people have asked how to prepare for translations ahead of built-in support, so while we have some design to do on how the feature will actually be implemented, here's the current thinking:

You'll configure translations for the system, then enable them on a list/field basis. Behind the scenes, fields will be duplicated so for a system with en, fr, de languages defined you'd see three fields for each translated field in a list. Imagine a Post with a translated title and content fields but non-translated slug field. The actual list would contain:

Post {
  title_en
  title_fr
  title_de
  content_en
  content_fr
  content_de
  slug
}

In the Admin UI, we'd have a select / tabs UI to let you switch between the different translations in the Edit Item form, and other UI for managing translations generally.

We'll probably look at ways to make querying for a translation through the GraphQL API easier too. This is the least well defined part of the plan at the moment, but I can imagine it looking something like this:

Post(where: { id: $id }, language: 'de') {
  title
  content
  slug
}

... and that would select the values from the German translation for the translated fields.

Right now, you can build (most of) this manually but we know that proper support will make it much easier to build multilingual apps with Keystone.

If you do want to go with a workaround, it's pretty easy to abstract this with a function when you're defining lists in Keystone Next:

const LANGUAGES = { en: 'English', fr: 'French', de: 'German' };

function translate(path, field) {
  const fields = {};
  for (const [key, label] of Object.entries(LANGUAGES)) {
    fields[`${path}_${key}`] = field;
  }
  return fields;
}

const Post = list({
  fields: {
    ...translate('title', text()),
    ...translate('content', text({ ui: { displayMode: 'textarea' })),
    slug: text({ isUnique: true }),
  },
});

I also want to shout out to @GasimGasimzada for a great and thorough write-up above, we'll be referring to that when we get to the design stage for this and flesh out the idea I outlined above properly.

If anybody else has other ideas or requirements to add here, that would also be really helpful for when we get to building this in the coming months.

[vonba]

@JedWatson that is great news, thank you! To me everything you described makes sense for my use cases. I think indeed having the ability to specify the language as part of the query would be super useful. I would also mention that any multilingual website that cares about SEO will also need slugs to be translatable. But I guess following your example above it would probably just mean adding several slug fields (slug_fr, slug_de, etc).

[JedWatson]

@vonba you're right about the slug field, was reaching for an example but something like a publishedStatus and author field probably make more sense for that! will keep in mind for the examples when we build them.

@gautamsi I'm expecting this would work with any field type; text, document, select, json etc. Is that what you meant?

[vonba]

@JedWatson Something like an e-commerce site might also want to display different images or other media for different locales/countries. I don't know if this is something that you would want to specifically want the API to deal with, but I think it's something that would come up.

[gautamsi]

@JedWatson I am selectively interested in automatic translation of document fields given block and json structure.

[acerspyro]

Is there any update on this?
I'm looking for a CMS to replace Wordpress for good and KeystoneJS looked perfect for doing that, until I ran into translations: as I'm from Austria it is necessary for me to deliver a backend in german and a frontend with different languages.

Are you guys planning to do something about this or will KeystoneJS remain a one language system?

Because in that case I'll need to (sadly) look somewhere else :(

Same, I'm moving away from WordPress and I've tried Strapi (which does have native l10n support), but I found its admin UI too limited for my uses.

I'm really hoping there's a strong focus on l10n in KeystoneJS 6 as it's otherwise much better than Strapi for my use case.

As a web developer in Quebec, no l10n is absolutely a no-go unless your client wants fines up the ass from the provincial government or wants to serve a Quebec-only, french-speaking only client base.

[sladg]

Do we have a timeline for this? Any predictions when translations might be available? :)

[stefanpl]

Just a single data point:

Not having a translation system built-in, and not having any solution that's ready to use, unfortunately disqualifies Keystone for us. We'd very much like to switch to a more modern CMS, but having to implement our own translation system before even being able to get started is a no-no.

Gonna check back in a couple of years, I guess …

[acerspyro]

If anybody else has other ideas or requirements to add here, that would also be really helpful for when we get to building this in the coming months.

FWIW, here are the absolutely necessary features I can think of:

• Fetching document by language
• Fetching an array of all configured languages
• Fetching an array of all available languages for the desired document
• Linked fields (i.e. same text across languages) and single-language documents that always return the same document regardless of the requested locale

And a nice-to-have:

• Permission to limit access to particular languages (for translator accounts, for example)

[ghost]

Awesome that this is on the roadmap!! This is the only reason why we haven't made the switch to Keystone yet, and we would really love to switch because the DX and workflow are so great!

For our use case we would need to support not just language codes like en but also full locales such as en-GB. This might simply be a matter of which exact strings we configure, but just throwing that out there :)

The approach you suggested for querying @JedWatson sounds ideal to me. This is also what we (and probably many others) are used to from the likes of Contentful and Strapi and it works very well in my opinion.

query {
  posts(locale: "en-GB") {
    title
  }
}

A super handy nice-to-have might be to mark a field to fall back to the default locale when empty. This way, you could for example provide the title of your post in all languages, while only supplying the image for the default locale, saving editors a bunch of time.

I would also like to 2nd @acerspyro's suggestion of locale-aware access control. I think we would be able to code that using the existing access control API without any need for changes, but just wanted to mention that it's a use-case we face quite often.

[adj87]

any progress on this??

[utoppo]

I also think multilingual support is fundamental and should be a top priority. Not having multilingual support means the CMS cannot be used for a lot of projects and many developers will have to look for another solution, although they really like the system in general.

There seems to be a lack of awareness of internationalization in projects (not only open source) developed and started in mainly english-speaking regions. Examples are WordPress, Webflow, shopify. Whereas projects started in non-english speaking countries like France, Germany, always implement strong multilingual support from the very beginning e.g. strapi, Contentful, neos.io

It seems like it gets harder and harder to add real multilingual support the longer a project exists and is developed (Looking at WordPress and webflow, where it has been promised for years, and probably will take another few years, probably because it wasn't thought of from the beginning)

So, the sooner the better, I think.

[troycura]

Hi,

Do we have an open source server now that supports multiple language?

Thank you.

[acomagu]

I know Strapi, Webiny, Payload, Directus and Sanity.

[bekir-arkulpa]

Any progress ? i also think that translatable fields are a must, we are gonna implement our own system for now but a in built system with graphql integration like in Payload CMS would be really nice to have. I think Its a pretty basic feature that every CMS should have.

[troycura]

Do we have an open source server now that supports multiple language?

[au-top]

This is my branch, which supports administration page in other languages at compile time

Before the official support of i18n , the project mainly adds the ability to display other languages to some text on the admin page

https://github.com/au-top/keystone

[acomagu]

FYI: Strapi addresses this issue by creating separate records for each language.

On the roadmap, Nested Fields has higher priority than multilingual support. When Nested Fields supported, we may be able to realize multilingual support by replacing text(translated) field by a simple component including multi-language text. e.g.

const lists = {
  TranslatedText: nested({
    fields: {
      en: text(),
      ja: text(),
    },
  }),
};

[onionhammer]

I'll probably be going with payload because of this..

[flexdinesh]

If Keystone doesn't work for your use case we highly recommend Payload CMS. It's great!

[acerspyro]

I personally use Strapi, haven't tried PayloadCMS!

[vitalijalbu]

try directus (nodejs) or craftmcs (php)