Restructuring for better separation of Tool packages.

[mirror_edk2.git] / Tools / CodeTools / TianoTools / Pccts / CHANGES_FROM_131.txt

CHANGES FROM 1.31\r
\r
This file contains the migration of PCCTS from 1.31 in the order that\r
changes were made.  1.32b7 is the last beta before full 1.32.\r
Terence Parr, Parr Research Corporation 1995.\r
\r
\r
======================================================================\r
1.32b1\r
Added Russell Quong to banner, changed banner for output slightly\r
Fixed it so that you have before / after actions for C++ in class def\r
Fixed bug in optimizer that made it sometimes forget to set internal\r
        token pointers.  Only showed up when a {...} was in the "wrong spot".\r
\r
======================================================================\r
1.32b2\r
Added fixes by Dave Seidel for PC compilers in 32 bit mode (config.h\r
and set.h).\r
\r
======================================================================\r
1.32b3\r
Fixed hideous bug in code generator for wildcard and for ~token op.\r
\r
from Dave Seidel\r
\r
   Added pcnames.bat\r
   1. in antlr/main.c: change strcasecmp() to stricmp()\r
\r
   2. in dlg/output.c: use DLEXER_C instead on "DLexer.C"\r
\r
   3. in h/PBlackBox.h: use <iostream.h> instead of <stream.h>\r
\r
======================================================================\r
1.32b4\r
When the -ft option was used, any path prefix screwed up\r
the gate on the .h files\r
\r
Fixed yet another bug due to the optimizer.\r
\r
The exception handling thing was a bit wacko:\r
\r
a : ( A B )? A B\r
  | A C\r
  ;\r
  exception ...\r
\r
caused an exception if "A C" was the input.  In other words,\r
it found that A C didn't match the (A B)? pred and caused\r
an exception rather than trying the next alt.  All I did\r
was to change the zzmatch_wsig() macros.\r
\r
Fixed some problems in gen.c relating to the name of token\r
class bit sets in the output.\r
\r
Added the tremendously cool generalized predicate.  For the\r
moment, I'll give this bried description.\r
\r
a : <<predicate>>? blah\r
  | foo\r
  ;\r
\r
This implies that (assuming blah and foo are syntactically\r
ambiguous) "predicate" indicates the semantic validity of\r
applying "blah".  If "predicate" is false, "foo" is attempted.\r
\r
Previously, you had to say:\r
\r
a : <<LA(1)==ID ? predicate : 1>>? ID\r
  | ID\r
  ;\r
\r
Now, you can simply use "predicate" without the ?: operator\r
if you turn on ANTLR command line option: "-prc on".  This\r
tells ANTLR to compute that all by itself.  It computes n\r
tokens of lookahead where LT(n) or LATEXT(n) is the farthest\r
ahead you look.\r
\r
If you give a predicate using "-prc on" that is followed\r
by a construct that can recognize more than one n-sequence,\r
you will get a warning from ANTLR.  For example,\r
\r
a : <<isTypeName(LT(1)->getText())>>? (ID|INT)\r
  ;\r
\r
This is wrong because the predicate will be applied to INTs\r
as well as ID's.  You should use this syntax to make\r
the predicate more specific:\r
\r
a : (ID)? => <<isTypeName(LT(1)->getText())>>? (ID|INT)\r
  ;\r
\r
which says "don't apply the predicate unless ID is the\r
current lookahead context".\r
\r
You cannot currently have anything in the "(context)? =>"\r
except sequences such as:\r
\r
( LPAREN ID | LPAREN SCOPE )? => <<pred>>?\r
\r
I haven't tested this THAT much, but it does work for the\r
C++ grammar.\r
\r
======================================================================\r
1.32b5\r
\r
Added getLine() to the ANTLRTokenBase and DLGBasedToken classes\r
left line() for backward compatibility.\r
----\r
Removed SORCERER_TRANSFORM from the ast.h stuff.\r
-------\r
Fixed bug in code gen of ANTLR such that nested syn preds work more\r
efficiently now.  The ANTLRTokenBuffer was getting very large\r
with nested predicates.\r
------\r
Memory leak is now gone from ANTLRTokenBuf; all tokens are deleted.\r
For backward compatibility reasons, you have to say parser->deleteTokens()\r
or mytokenbuffer->deleteTokens() but later it will be the default mode.\r
Say this after the parser is constructed. E.g.,\r
\r
    ParserBlackBox<DLGLexer, MyParser, ANTLRToken> p(stdin);\r
    p.parser()->deleteTokens();\r
    p.parser()->start_symbol();\r
\r
\r
==============================\r
1.32b6\r
\r
Changed so that deleteTokens() will do a delete ((ANTLRTokenBase *))\r
on the ptr.  This gets the virtual destructor.\r
\r
Fixed some weird things in the C++ header files (a few return types).\r
\r
Made the AST routines correspond to the book and SORCERER stuff.\r
\r
New token stuff:  See testcpp/14/test.g\r
\r
ANTLR accepts a #pragma gc_tokens which says\r
[1]     Generate label = copy(LT(1)) instead of label=LT(1) for\r
        all labeled token references.\r
[2]     User now has to define ANTLRTokenPtr (as a class or a typedef\r
        to just a pointer) as well as the ANTLRToken class itself.\r
		See the example.\r
\r
To delete tokens in token buffer, use deleteTokens() message on parser.\r
\r
        All tokens that fall off the ANTLRTokenBuffer get deleted\r
        which is what currently happens when deleteTokens() message\r
        has been sent to token buffer.\r
\r
We always generate ANTLRTokenPtr instead of 'ANTLRToken *' now.\r
Then if no pragma set, ANTLR generates a\r
\r
        class ANTLRToken;\r
        typedef ANTLRToken *ANTLRTokenPtr;\r
\r
in each file.\r
\r
Made a warning for x:rule_ref <<$x>>; still no warning for $i's, however.\r
class BB {\r
\r
a : x:b y:A <<$x\r
$y>>\r
  ;\r
\r
b : B;\r
\r
}\r
generates\r
Antlr parser generator   Version 1.32b6   1989-1995\r
test.g, line 3: error: There are no token ptrs for rule references: '$x'\r
\r
===================\r
1.32b7:\r
\r
[With respect to token object garbage collection (GC), 1.32b7\r
 backtracks from 1.32b6, but results in better and less intrusive GC.\r
 This is the last beta version before full 1.32.]\r
\r
BIGGEST CHANGES:\r
\r
o	The "#pragma gc_tokens" is no longer used.\r
\r
o	.C files are now .cpp files (hence, makefiles will have to\r
	be changed; or you can rerun genmk).  This is a good move,\r
	but causes some backward incompatibility problems.  You can\r
	avoid this by changing CPP_FILE_SUFFIX to ".C" in pccts/h/config.h.\r
\r
o	The token object class hierarchy has been flattened to include\r
	only three classes: ANTLRAbstractToken, ANTLRCommonToken, and\r
	ANTLRCommonNoRefCountToken.  The common token now does garbage\r
	collection via ref counting.\r
\r
o	"Smart" pointers are now used for garbage collection.  That is,\r
	ANTLRTokenPtr is used instead of "ANTLRToken *".\r
\r
o	The antlr.1 man page has been cleaned up slightly.\r
\r
o	The SUN C++ compiler now complains less about C++ support code.\r
\r
o	Grammars which subclass ANTLRCommonToken must wrap all token\r
	pointer references in mytoken(token_ptr).  This is the only\r
	serious backward incompatibility.  See below.\r
\r
\r
MINOR CHANGES:\r
\r
--------------------------------------------------------\r
1	deleteTokens()\r
\r
The deleteTokens() message to the parser or token buffer has been changed\r
to one of:\r
\r
    void noGarbageCollectTokens()   { inputTokens->noGarbageCollectTokens(); }\r
    void garbageCollectTokens()     { inputTokens->garbageCollectTokens(); }\r
\r
The token buffer deletes all non-referenced tokens by default now.\r
\r
--------------------------------------------------------\r
2	makeToken()\r
\r
The makeToken() message returns a new type.  The function should look\r
like:\r
\r
    virtual ANTLRAbstractToken *makeToken(ANTLRTokenType tt,\r
                                          ANTLRChar *txt,\r
                                          int line)\r
    {\r
        ANTLRAbstractToken *t = new ANTLRCommonToken(tt,txt);\r
        t->setLine(line);\r
        return t;\r
    }\r
\r
--------------------------------------------------------\r
3	TokenType\r
\r
Changed TokenType-> ANTLRTokenType  (often forces changes in AST defs due\r
to #[] constructor called to AST(tokentype, string)).\r
\r
--------------------------------------------------------\r
4	AST()\r
\r
You must define AST(ANTLRTokenPtr t) now in your AST class definition.\r
You might also have to include ATokPtr.h above the definition; e.g.,\r
if AST is defined in a separate file, such as AST.h, it's a good idea\r
to include ATOKPTR_H (ATokPtr.h).  For example,\r
\r
	#include ATOKPTR_H\r
	class AST : public ASTBase {\r
	protected:\r
	    ANTLRTokenPtr token;\r
	public:\r
	    AST(ANTLRTokenPtr t) { token = t; }\r
	    void preorder_action() {\r
	        char *s = token->getText();\r
	        printf(" %s", s);\r
	    }\r
	};\r
\r
Note the use of smart pointers rather than "ANTLRToken *".\r
\r
--------------------------------------------------------\r
5	SUN C++\r
\r
From robertb oakhill.sps.mot.com Bob Bailey. Changed ANTLR C++ output\r
to avoid an error in Sun C++ 3.0.1.  Made "public" return value\r
structs created to hold multiple return values public.\r
\r
--------------------------------------------------------\r
6	genmk\r
\r
Fixed genmk so that target List.* is not included anymore.  It's\r
called SList.* anyway.\r
\r
--------------------------------------------------------\r
7	\r vs \n\r
\r
Scott Vorthmann <vorth cmu.edu> fixed antlr.g in ANTLR so that \r\r
is allowed as the return character as well as \n.\r
\r
--------------------------------------------------------\r
8	Exceptions\r
\r
Bug in exceptions attached to labeled token/tokclass references.  Didn't gen\r
code for exceptions.  This didn't work:\r
\r
a : "help" x:ID\r
  ;\r
        exception[x]\r
        catch MismatchedToken : <<printf("eh?\n");>>\r
\r
Now ANTLR generates (which is kinda big, but necessary):\r
\r
        if ( !_match_wsig(ID) ) {\r
                if ( guessing ) goto fail;\r
                _signal=MismatchedToken;\r
                switch ( _signal ) {\r
                case MismatchedToken :\r
                        printf("eh?\n");\r
                        _signal = NoSignal;\r
                        break;\r
                default :\r
                        goto _handler;\r
                }\r
        }\r
\r
which implies that you can recover and continue parsing after a missing/bad\r
token reference.\r
\r
--------------------------------------------------------\r
9	genmk\r
\r
genmk now correctly uses config file for CPP_FILE_SUFFIX stuff.\r
\r
--------------------------------------------------------\r
10	general cleanup / PURIFY\r
\r
Anthony Green <green vizbiz.com> suggested a bunch of good general\r
clean up things for the code; he also suggested a few things to\r
help out the "PURIFY" memory allocation checker.\r
\r
--------------------------------------------------------\r
11	$-variable references.\r
\r
Manuel ORNATO indicated that a $-variable outside of a rule caused\r
ANTLR to crash.  I fixed this.\r
\r
12	Tom Moog suggestion\r
\r
Fail action of semantic predicate needs "{}" envelope.  FIXED.\r
\r
13	references to LT(1).\r
\r
I have enclosed all assignments such as:\r
\r
             _t22 = (ANTLRTokenPtr)LT(1);\r
\r
in "if ( !guessing )" so that during backtracking the reference count\r
for token objects is not increased.\r
\r
\r
TOKEN OBJECT GARBAGE COLLECTION\r
\r
1	INTRODUCTION\r
\r
The class ANTLRCommonToken is now garbaged collected through a "smart"\r
pointer called ANTLRTokenPtr using reference counting.  Any token\r
object not referenced by your grammar actions is destroyed by the\r
ANTLRTokenBuffer when it must make room for more token objects.\r
Referenced tokens are then destroyed in your parser when local\r
ANTLRTokenPtr objects are deleted.  For example,\r
\r
a : label:ID ;\r
\r
would be converted to something like:\r
\r
void yourclass::a(void)\r
{\r
	zzRULE;\r
	ANTLRTokenPtr label=NULL;	// used to be ANTLRToken *label;\r
        zzmatch(ID);\r
        label = (ANTLRTokenPtr)LT(1);\r
	consume();\r
	...\r
}\r
\r
When the "label" object is destroyed (it's just a pointer to your\r
input token object LT(1)), it decrements the reference count on the\r
object created for the ID.  If the count goes to zero, the object\r
pointed by label is deleted.\r
\r
To correctly manage the garbage collection, you should use\r
ANTLRTokenPtr instead of "ANTLRToken *".  Most ANTLR support code\r
(visible to the user) has been modified to use the smart pointers.\r
\r
***************************************************************\r
Remember that any local objects that you create are not deleted when a\r
lonjmp() is executed.  Unfortunately, the syntactic predicates (...)?\r
use setjmp()/longjmp().  There are some situations when a few tokens\r
will "leak".\r
***************************************************************\r
\r
2	DETAILS\r
\r
o	The default is to perform token object garbage collection.\r
	You may use parser->noGarbageCollectTokens() to turn off\r
	garbage collection.\r
\r
\r
o	The type ANTLRTokenPtr is always defined now (automatically).\r
	If you do not wish to use smart pointers, you will have to\r
	redefined ANTLRTokenPtr by subclassing, changing the header\r
	file or changing ANTLR's code generation (easy enough to\r
	do in gen.c).\r
\r
o	If you don't use ParserBlackBox, the new initialization sequence is:\r
\r
	    ANTLRTokenPtr aToken = new ANTLRToken;\r
	    scan.setToken(mytoken(aToken));\r
\r
	where mytoken(aToken) gets an ANTLRToken * from the smart pointer.\r
\r
o	Define C++ preprocessor symbol DBG_REFCOUNTTOKEN to see a bunch of\r
	debugging stuff for reference counting if you suspect something.\r
\r
\r
3	WHY DO I HAVE TO TYPECAST ALL MY TOKEN POINTERS NOW??????\r
\r
If you subclass ANTLRCommonToken and then attempt to refer to one of\r
your token members via a token pointer in your grammar actions, the\r
C++ compiler will complain that your token object does not have that\r
member.  For example, if you used to do this\r
\r
<<\r
class ANTLRToken : public ANTLRCommonToken {\r
        int muck;\r
	...\r
};\r
>>\r
\r
class Foo {\r
a : t:ID << t->muck = ...; >> ;\r
}\r
\r
Now, you must do change the t->muck reference to:\r
\r
a : t:ID << mytoken(t)->muck = ...; >> ;\r
\r
in order to downcast 't' to be an "ANTLRToken *" not the\r
"ANTLRAbstractToken *" resulting from ANTLRTokenPtr::operator->().\r
The macro is defined as:\r
\r
/*\r
 * Since you cannot redefine operator->() to return one of the user's\r
 * token object types, we must down cast.  This is a drag.  Here's\r
 * a macro that helps.  template: "mytoken(a-smart-ptr)->myfield".\r
 */\r
#define mytoken(tp) ((ANTLRToken *)(tp.operator->()))\r
\r
You have to use macro mytoken(grammar-label) now because smart\r
pointers are not specific to a parser's token objects.  In other\r
words, the ANTLRTokenPtr class has a pointer to a generic\r
ANTLRAbstractToken not your ANTLRToken; the ANTLR support code must\r
use smart pointers too, but be able to work with any kind of\r
ANTLRToken.  Sorry about this, but it's C++'s fault not mine.  Some\r
nebulous future version of the C++ compilers should obviate the need\r
to downcast smart pointers with runtime type checking (and by allowing\r
different return type of overridden functions).\r
\r
A way to have backward compatible code is to shut off the token object\r
garbage collection; i.e., use parser->noGarbageCollectTokens() and\r
change the definition of ANTLRTokenPtr (that's why you get source code\r
<wink>).\r
\r
\r
PARSER EXCEPTION HANDLING\r
\r
I've noticed some weird stuff with the exception handling.  I intend\r
to give this top priority for the "book release" of ANTLR.\r
\r
==========\r
1.32 Full Release\r
\r
o	Changed Token class hierarchy to be (Thanks to Tom Moog):\r
\r
        ANTLRAbstractToken\r
          ANTLRRefCountToken\r
             ANTLRCommonToken\r
          ANTLRNoRefCountCommonToken\r
\r
o	Added virtual panic() to ANTLRAbstractToken.  Made ANTLRParser::panic()\r
	virtual also.\r
\r
o	Cleaned up the dup() stuff in AST hierarchy to use shallowCopy() to\r
	make node copies.  John Farr at Medtronic suggested this.  I.e.,\r
	if you want to use dup() with either ANTLR or SORCERER or -transform\r
	mode with SORCERER, you must defined shallowCopy() as:\r
\r
	virtual PCCTS_AST *shallowCopy()\r
	{\r
	    return new AST;\r
	    p->setDown(NULL);\r
	    p->setRight(NULL);\r
	    return p;\r
	}\r
\r
	or\r
\r
	virtual PCCTS_AST *shallowCopy()\r
	{\r
	    return new AST(*this);\r
	}\r
	\r
	if you have defined a copy constructor such as\r
\r
	AST(const AST &t)	// shallow copy constructor\r
	{\r
		token = t.token;\r
		iconst = t.iconst;\r
		setDown(NULL);\r
		setRight(NULL);\r
	}\r
\r
o	Added a warning with -CC and -gk are used together.  This is broken,\r
	hence a warning is appropriate.\r
\r
o	Added warning when #-stuff is used w/o -gt option.\r
\r
o	Updated MPW installation.\r
\r
o	"Miller, Philip W." <MILLERPW f1groups.fsd.jhuapl.edu> suggested\r
	that genmk be use RENAME_OBJ_FLAG RENAME_EXE_FLAG instead of\r
	hardcoding "-o" in genmk.c.\r
\r
o	made all exit() calls use EXIT_SUCCESS or EXIT_FAILURE.\r
\r
===========================================================================\r
1.33\r
\r
EXIT_FAILURE and EXIT_SUCCESS were not always defined.  I had to modify\r
a bunch of files to use PCCTS_EXIT_XXX, which forces a new version.  Sorry\r
about that.\r
\r