Recompiling changes to the console parser grammar

In the torque/engine/console folder, these are the files that are used to make the console parser:

- scan.l : tokenizer definition
- gram.y : parser definition
- yylex.c : core tokenizing engine from the MKS parser tools
- yyparse.c : MKS core parser engine
- scan.cc : the tokenizer, generated from scan.l and yylex.c by MKS lex
- gram.cc : the parser, generated from gram.y and yyparse.c by MKS yacc
- gram.h : the token definitions, generated from gram.y by MKS yacc

The scan.cc, gram.cc, and gram.h files are the only ones that are "seen" by the VC++ projects and the makefiles. If you were to make changes to scan.l or gram.y, the projects/makefiles would not know that a rebuild was needed, and they couldn't anyway unless you possess the MKS parser tools.

Fortunately scan.l and gram.y can be processed by other tools. I'm assuming that when the TGE repository was created, scan.l and gram.y were up-to-date compared to the generated files that were provided. And I make no promises about the efficiency of the generated parser compared to that of the MKS tools. But I have rebuilt the console parser using non-MKS tools, and everything seems to work so far.

The changes described below are for rebuilding the .l and .y files using Cygwin flex and bison. I wouldn't be surprised if other versions of flex/bison would be happy with exactly these same changes, or nearly so. For other tools, though, you may have to experiment a bit.


1) Rename scan.l to CMD.l, and gram.y to CMD.y. You can delete or move away all other files listed above; you won't be needing them for the non-MKS build.

2) In CMD.l, fix the include file paths; they should be:

#include "platform/platform.h"
#include "console/console.h"
#include "console/ast.h"
#include "console/gram.h"
#include "core/stringTable.h"

3) In CMD.l, change the include of "console/gram.h" to include "console/CMD.tab.h" instead.

4) In CMD.l, under the "Prototypes" section near the top, add:

int CMDgetc();
#define YY_INPUT(buf,result,max_size) \
{ \
	int c = '*', n; \
	for ( n = 0; n < max_size && \
		     (c = CMDgetc()) != EOF && c != '\n'; ++n ) \
		buf[n] = (char) c; \
	if ( c == '\n' ) \
		buf[n++] = (char) c; \
	result = n; \
}

5) In CMD.y, fix the include file paths; they should be:

#include "console/console.h"
#include "console/ast.h"
#include <stdlib.h>
#include <stdio.h>
#include "console/consoleInternal.h"

6) In CMD.y, fix the definition of YYDEBUG; it should be:

#ifndef YYDEBUG
#define YYDEBUG 1
#endif

7) In CMD.y, under the define of YY_ARGS, add:

int yylex();
void yyerror(char * s, ...);

8) In compiler.cc, add "#include " above the other includes.

9) In compiler.cc, change the include of "console/gram.h" to include "console/CMD.tab.h" instead.

10) In compiler.cc, change "extern void CMD_reset();" to "extern void CMDrestart(FILE *input_file);"

11) In compiler.cc, change all occurrences of "CMD_reset();" to "CMDrestart(NULL);"

12) In compiledEval.cc, change the include of "console/gram.h" to include "console/CMD.tab.h" instead.

13) Start up VC++ in the Torque SDK workspace, and select the Torque Demo project.

14) In the console folder, remove gram.h, gram.cc, and scan.cc from the list of source files (for each filename, select it and hit the delete key).

15) Now follow the HowTo on Incorporating a parser into your project, using CMD.l and CMD.y. There are only a few special things to be aware of. First, do not set the noyywrap option in CMD.l (the never-interactive option is still needed though). Second, when you're setting up the command-line options for the tokenizer to process CMD.l, do not choose to make it case-insensitive; if you do, the key name "tab" will be confused with the string operator "TAB" (and maybe other problems). Finally, if you adjust the makefiles system, the relevant targets file for steps 2 and 3 of "Makefile setup" is torque/engine/targets.torque.mk; for step 2 of "Makefile setup", in that file you should replace console/scan.cc and console/gram.cc with console/CMD.cc and console/CMD.tab.cc.

16) Repeat steps 14 & 15 for the Torque Lib project, since it also builds the console parser.

There's one more consideration: scan.cc has had some changes (from version 1.2 to 1.3) that were made directly to it, rather than making changes to scan.l and then regenerating scan.cc. Fortunately it's rather easy to assimilate those changes into CMD.l, so that they will be included in the parser you generate.

Note: TGE licensees can download a zipfile of the new CMD.l and CMD.y from a thread on the private forums.