Bring languages into the tree (alliedmodders#1625)

* translations: bring languages into tree

* Update translation phrases changed since 2021

* Update packaging script to include all translations

* Update languages.cfg

* Add Latin American Spanish translations

This is a copy of spanish for now.

* Ignore "en" when looking for translation folders

English is the default and doesn't use a subfolder.

* Only add each translation folder once

Korean "ko" is in there twice.

* Compare language coverage to english

All phrases are compared to the english baseline files and any differences
are reported. The differences are pushed to a Github Project as well for
an easier overview.

Thank you to @nosoop for sharing the Python SMC parser!

* Add link to README

---------

Co-authored-by: Peace-Maker <[email protected]>

Author: KyleSanderson

.github/workflows/translations.yml

@@ -0,0 +1,38 @@
+name: Update translation project
+on:
+  push:
+    branches:
+      - master
+    paths:
+      - 'translations/**'
+jobs:
+  update_translations:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-python@v4
+        name: Setup Python 3.10
+        with:
+          python-version: 3.10
+
+      - name: Install Python dependencies
+        working-directory: tools/language_check
+        run: |
+          python -m pip install --upgrade -r requirements.txt
+
+      - name: Generate token
+        id: generate_token
+        uses: tibdex/github-app-token@36464acb844fc53b9b8b2401da68844f6b05ebb0
+        with:
+          app_id: ${{ secrets.APP_ID }}
+          private_key: ${{ secrets.APP_PEM }}
+
+      - name: Get project data
+        working-directory: tools/language_check
+        env:
+          GITHUB_TOKEN: ${{ steps.generate_token.outputs.token }}
+          ORGANIZATION: alliedmodders
+          PROJECT_NUMBER: 1
+        run: |
+          python ./compare_translation_phrases.py 
+
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 

README.md

@@ -16,6 +16,7 @@ Development
 - [SourcePawn scripting](https://wiki.alliedmods.net/Category:SourceMod_Scripting): SourcePawn examples and introduction to the language
 - [SourceMod plugin API](https://sm.alliedmods.net/new-api): Online SourceMod plugin API reference generated from the include files
 - [SourceMod extension development](https://wiki.alliedmods.net/Category:SourceMod_Development): C++ examples and introduction to various extension interfaces
+- [Translation project](https://github.com/orgs/alliedmodders/projects/1): Help [translate SourceMod](https://wiki.alliedmods.net/Translations_(SourceMod_Scripting)) into your language
 
 Contact
 -------

configs/languages.cfg

@@ -1,4 +1,37 @@
 "Languages"
 {
-	"en"			"English"
+	"en"		"English"		// English
+	"ar"		"Arabic"		// Arabic
+	"pt"		"Brazilian"		// Brazilian Portuguese
+	"bg"		"Bulgarian"		// Bulgarian
+	"cze"		"Czech"		// Czech
+	"da"		"Danish"		// Danish
+	"nl"		"Dutch"		// Dutch
+	"fi"		"Finnish"		// Finnish
+	"fr"		"French"		// French
+	"de"		"German"		// German
+	"el"		"Greek"		// Greek
+	"he"		"Hebrew"		// Hebrew
+	"hu"		"Hungarian"		// Hungarian
+	"it"		"Italian"		// Italian
+	"jp"		"Japanese"		// Japanese
+	"ko"		"KoreanA"		// Korean
+	"ko"		"Korean"		// Korean
+	"las"		"LatAm"		// Latin American Spanish
+	"lv"		"Latvian"		// Latvian
+	"lt"		"Lithuanian"		// Lithuanian
+	"no"		"Norwegian"		// Norwegian
+	"pl"		"Polish"		// Polish
+	"pt_p"		"Portuguese"		// Portuguese
+	"ro"		"Romanian"		// Romanian
+	"ru"		"Russian"		// Russian
+	"chi"		"SChinese"		// Chinese (Simplified)
+	"sk"		"Slovak"		// Slovak
+	"es"		"Spanish"		// Spanish
+	"sv"		"Swedish"		// Swedish
+	"zho"		"TChinese"		// Chinese (Traditional)
+	"th"		"Thai"		// Thai
+	"tr"		"Turkish"		// Turkish
+	"ua"		"Ukrainian"		// Ukrainian
+	"vi"		"Vietnamese"		// Vietnamese
 }

tools/buildbot/PackageScript

@@ -1,5 +1,6 @@
 # vim: set ts=8 sts=2 sw=2 tw=99 et ft=python:
 import os
+import re
 
 builder.SetBuildFolder('package')
 
@@ -126,3 +127,17 @@ helpers.CopyFiles('plugins/basevotes', 'addons/sourcemod/scripting/basevotes')
 helpers.CopyFiles('plugins/basebans', 'addons/sourcemod/scripting/basebans')
 helpers.CopyFiles('plugins/funcommands', 'addons/sourcemod/scripting/funcommands')
 helpers.CopyFiles('plugins/playercommands', 'addons/sourcemod/scripting/playercommands')
+
+with open(os.path.join(builder.sourcePath, 'configs/languages.cfg'), 'r') as f:
+  language_re = re.compile(r'^\s*"([^"]+)"\s+"[^"]+"')
+  added_languages = set(["en"])
+  for line in f.read().splitlines():
+    match = language_re.match(line)
+    if match:
+      lang_code = match.group(1)
+      if lang_code in added_languages:
+        continue
+      output_path = os.path.join('addons/sourcemod/translations', lang_code)
+      helpers.CreateFolders([output_path])
+      helpers.CopyFiles(os.path.join('translations', lang_code), output_path)
+      added_languages.add(lang_code)

tools/buildbot/package.pl

@@ -36,27 +36,6 @@
 #Switch to the output folder.
 chdir(Build::PathFormat('../../../OUTPUT/package'));
 
-print "Downloading languages.cfg...\n";
-# Don't check certificate. It will fail on the slaves and we're resolving to internal addressing anyway
-system('wget --no-check-certificate -q -O addons/sourcemod/configs/languages.cfg "https://sm.alliedmods.net/translator/index.php?go=translate&op=export_langs"');
-open(my $fh, '<', 'addons/sourcemod/configs/languages.cfg')
-    or die "Could not open languages.cfg' $!";
- 
-while (my $ln = <$fh>) {
-    if ($ln =~ /"([^"]+)"\s*"[^"]+.*\((\d+)\) /)
-    {
-	my $abbr = $1;
-	my $id = $2;
-
-	print "Downloading language pack $abbr.zip...\n";
-        # Don't check certificate. It will fail on the slaves and we're resolving to internal addressing anyway
-        system("wget --no-check-certificate -q -O $abbr.zip \"https://sm.alliedmods.net/translator/index.php?go=translate&op=export&lang_id=$id\"");
-        system("unzip -qo $abbr.zip -d addons/sourcemod/translations/");
-        unlink("$abbr.zip");
-    }
-}
-close($fh);
-
 unless (-e '../GeoLite2-City_20191217.tar')
 {
     print "Downloading GeoLite2-City.mmdb...\n";