Where is Langc for localization?

#1 I don't think they got around to including it in the betas, but I believe you can get it off TGEA.

#2 I didn't find it in TGEA.
Do you have it? Does someone have this code?

Or do you have other solution to manage localization?

#3 In our last TGEA 1.7.1 project we had multiple languages working. I'm not sure if we had to do a few code changes to make it work (I'm sure we also added support for localized bitmaps), but we sure had langc, lso files and GuiControl's langTable working properly. I'll have to shuffle through my old backups to get to the project's source, so it'll take a while.

#4 Alright, I gave it a look. The language support is a tad different in T3D, but it's hooked into the most important places already. The only problem is that the language table format changed a bit, and the LSO files created by langc don't work on it.

The good news is that it's pretty simple to create add some source code so you can compile your language tables directly from T3D, so you don't need langc. I also threw in a change to allow longer strings (up to 2048 bytes - texts bigger than this should be ideally stored in separate text files read using FileObject only when you need to display them, to keep RAM usage low).

Open source/i18n/lang.cpp. After this line:

#include "i18n/lang.h"

Add this:

#include "core/fileObject.h"
#include "core/util/str.h"
#include "core/strings/unicode.h"

Now replace this:

bool LangFile::load(Stream *s)
{
	freeTable();
	
	while(s->getStatus() != Stream::EOS)
	{
		char buf[256];
		s->readString(buf);
		addString((const UTF8*)buf);
	}
	return true;
}

By this:

bool LangFile::load(Stream *s)
{
	freeTable();
	
	while(s->getStatus() == Stream::Ok)
	{
		char buf[2048];
      s->readLongString(2048,buf);
      if (s->getStatus() == Stream::Ok)
         addString((const UTF8*)buf);
	}
	return true;
}

Replace this:

U32 i;
for(i = 0;i < mStringTable.size();i++)
{
	s->writeString((char*)mStringTable[i]);
}

By this:

U32 i;
for(i = 0;i < mStringTable.size();i++)
{
  s->writeLongString(2048, (char*)mStringTable[i]);
}

And finally add this at the bottom of the file:

ConsoleFunction( CompileLanguage, void, 2, 3, "(string inputFile, [bool createMap]) Compiles a LSO language file."
                                              " if createIndex is true, will also create languageMap.cs with"
                                              " the global variables for each string index."
                                              " The input file must follow this example layout:"
                                              " TXT_HELLO_WORLD = Hello world in english!" )
{
   UTF8 scriptFilenameBuffer[1024];	
	Con::expandScriptFilename((char*)scriptFilenameBuffer, sizeof(scriptFilenameBuffer), argv[1]);

   if(!Torque::FS::IsFile(scriptFilenameBuffer))   
   {
      Con::errorf("CompileLanguage - file %s not found", scriptFilenameBuffer);
      return;
   }

   FileObject file;
   if (!file.readMemory(scriptFilenameBuffer))
   {
      Con::errorf("CompileLanguage - couldn't read file %s", scriptFilenameBuffer);
      return;
   }

   bool createMap = argc > 2 ? dAtob(argv[2]) : false;
   FileStream *mapStream = NULL;	
   if (createMap)
   {
      Torque::Path mapPath = scriptFilenameBuffer;
      mapPath.setFileName("languageMap");
      mapPath.setExtension("cs");
      if((mapStream = FileStream::createAndOpen( mapPath, Torque::FS::File::Write )) == NULL)
         Con::errorf("CompileLanguage - failed creating languageMap.cs");         
   }

   LangFile langFile;
   const U8* inLine = NULL;
   const char* separatorStr = " = ";
   S32 stringId = 0;
   while( (inLine = file.readLine())[0] != 0  )
	{	     
      char* line;
      chompUTF8BOM( (const char *)inLine, &line );
      char* div = dStrstr(line, separatorStr);
      if (div == NULL)
      {
         Con::errorf("Separator %s not found in line: %s", separatorStr, line);
         Con::errorf("Could not determine string name ID");
         continue;
      }
      *div = 0;      
      char* text = div + dStrlen(separatorStr);

      langFile.addString((const UTF8*)text);

      if (mapStream)
      {
         String mapLine = String::ToString("$%s = %i;", line, stringId);    
         mapStream->writeLine((const U8*)mapLine.c_str());
         String commentLine = String::ToString("// %s", text);    
         mapStream->writeLine((const U8*)commentLine.c_str());
      }

      stringId++;
	}

   Torque::Path lsoPath = scriptFilenameBuffer;   
   lsoPath.setExtension("lso");
   langFile.save(lsoPath.getFullPath());

   if (mapStream)
      delete mapStream;
}

Continues in the post below...

#5 CompileLanguage() does exactly what langc.exe did: it takes a text file and compiles a LSO which can be loaded into a langTable. It can also generate a languageMap.cs file, which contains the global variables with the indices for each string. Here's an example of how to use it:

1) Create a file called "english.txt" in game/scripts, paste this inside it:

txt_hello = Hello world
txt_language = This is the english language
txt_last = This is the last string

2) Add the code below at the bottom of game/scripts/main.cs:

new LangTable(mainLangTable);
$I18N::default = mainLangTable.getId();

// This should only be done when the .txt is newer than the .lso
compileLanguage("./english.txt", 1);

mainLangTable.addLanguage("./english.lso", "English");
mainLangTable.setCurrentLanguage(0);
exec("./languageMap.cs");

function switchLanguage(%language)
{
   mainLangTable.setCurrentLanguage(%language);
   Canvas.setContent(Canvas.getContent());      
}

And in game/main.cs, in the CreateCanvas() function, add this before the return:

Canvas.langTableMod = "default";

Now run T3D, open the GUI editor, create a button and type "txt_hello" this in the textID field (without the quotes). You should see the localized string there. The script function switchLanguage() I provided is an easy way to switch languages. Here's a test switching between English and Japanese (I had to change the font in the GuiButtonProfile since Arial doesn't have Japanese characters - I used Meiryo UI):

English:
Japanese:

#6 Many thanks for this.

I will test that this week.

I think you can deliver it as a resource, some other people might be interested by this functionality.

#7 I confirmed this is working perfectly. thanks again!

You should deliver this as a resource or better GG should add it in the next version / OIfficial documentation.

Just a quick question on what you did to understand:
1/ To update the GUI I just need to set text_id
2/ To use it in the scripts, this is where the .cs (map file) is useful and I need to call mainlangtable.getstring($txt_hello) to get it.
3/ in C++?

in "Separator \"%s\" not" it was missing the \ due to the forum.

#8

Quote:1/ To update the GUI I just need to set text_id

Yes, when using localization you use "text_id" instead of "text". It is a number, not a string, so in order to have human-readable IDs we use the languageMap.cs file to generate global variables with each ID.

Quote:2/ To use it in the scripts, this is where the .cs (map file) is useful and I need to call mainlangtable.getstring($txt_hello) to get it.

Yeah, I forgot to tell about that. If you need a localized string in script, you need to grab it directly from the LangTable.

Quote:3/ in C++?

You'd need to do like the GUIs do it: get the LangTable and call getString() on it, but it's getting convoluted so maybe we need an easy-to-use function to easily grab strings, no? I can't do it right now, but I should be able to come up with one tomorrow or something like that. A good getLangString() function should support receiving the text_id variable names and resolve them internally.

#9 There is a resource by Thomas Huehn that does the same. I haven't been able to integrate either Manoel's changes or Thomas' resource, but the idea that you don't have to keep track of IDs is a very appealing one.

I know it can get very tedious with too many strings. Perhaps the built in system should also use a default language to match strings. It's more convenient to develop that way.

#10 I think we have 2 different approach between Manoel & Thomas.

Thomas is looking for a string to get the translation. That's mean if you need to change the "default" value, you have to change it everywhere :(
in Manoel approach, you have global variable that are created for each table entry, so you can call what ever you want by using them in TextId.
If you need to change text, all you have to do is in the lang.lso file. The only downside I can see, might be the number of Global variable that might be created once your game is completed.

Thomas is based on Langc.exe, but the exe is no more in T3D for reason that I don't know, so without it, you can't use Thomas resources :(

In my opinion, GG should take the lead on this topic and propose a solution. They had one previously, but they seems to have changed their mind as now we don't have the tool to use it but 90% of the code is still in the source.
May be Konrad, as an associate you can point this out and share what is their answer?

@Manoel,
In Thomas resource, we have some interesting function, this might be interesting to get them?
there is also the bug on LangFile::~LangFile()

#11 @elvince:

Tom Bampton, a fellow Torque 3D developer, who might be the author of the langc tool would probably know more about its discontinued support than anyone else. Let's hope this thread catches his attention.

Or it could be Christopher Evans who was last seen around here a year ago - which would explain the discontinued support.

Both of them contributed much to related TDN documentation.

I'm not familiar with the tool at all, but running through all the forum posts connected to the tool tells me that there should be an even better way of solving this problem.

#12 @elvince
In T3D the LangFile::~LangFile() bug is already fixed ;)

When you get the license for T3D you also have TGEA and TGE so you could get the langc from this packages. I thought about integrating the langc functionality in my tool, but dropped the idea because it would break the license, I guess.

While I didn't want to work with the Id's it's still possible. I've added the Tr([id]) command to the script, which translate the id and T([string]) translate the default language string into the current language.

What I not understand is:

Quote:That's mean if you need to change the "default" value, you have to change it everywhere :(

If you change the text in script you only need to change it once again in the default language (copy&paste). You can also use a dummy default language which is never displayed - a slang for example, so you only have to change the translation text.
The downside of text translation is, it need a bit more cpu power because it search the string and return the translation. I prefer it because, I'm to lazy to work with id's ;)

#13 "My" method isn't actually mine, that's how it came in TGEA. However, having the IDs as global variables is almost the same as having actual names, since you still have names.

#14 @Thomas,
I agree that I can download TGEA/TGE with T3D license. But there might be a reason to not put the langc.exe in it? It's very strange that I need to go back in old version to get tools.

My point on "default" value text is that, if you have a text that it is used in all of your gui. If your default text is really a language, then you have to change it everywhere.
If you want to have something that mean something to you, I suggest that you look at the global variable that we are using in "Manoel/TGEA" method.
Each ids are mapped in a "$XXX" variable so the only changes you are making is in the lso/lang file.

Maybe langc do other thing that this solution do not, but clearly what Manoel suggest with his solution is really interesting as it is fully integrated in your engine.

from your point of view, what's missing in this proposal. I'm sure both proposal can be mixed to get an improved solution.

#15 @Elvince: the console function I wrote does everything langc.exe did: it creates the LSO file and a .CS file with the global variable IDs. It's just the format of the .LSO that is slightly different, but the LSO files were just a straight list of UTF8 strings.