--- /dev/null
+\r
+ ------------------------------------------------------------\r
+ This is the second part of a two part file.\r
+ This is a list of changes to pccts 1.33 prior to MR13\r
+ For more recent information see CHANGES_FROM_133.txt\r
+ ------------------------------------------------------------\r
+\r
+ DISCLAIMER\r
+\r
+ The software and these notes are provided "as is". They may include\r
+ typographical or technical errors and their authors disclaims all\r
+ liability of any kind or nature for damages due to error, fault,\r
+ defect, or deficiency regardless of cause. All warranties of any\r
+ kind, either express or implied, including, but not limited to, the\r
+ implied warranties of merchantability and fitness for a particular\r
+ purpose are disclaimed.\r
+\r
+\r
+#153. (Changed in MR12b) Bug in computation of -mrhoist suppression set\r
+\r
+ Consider the following grammar with k=1 and "-mrhoist on":\r
+\r
+ r1 : (A)? => ((p>>? x /* l1 */\r
+ | r2 /* l2 */\r
+ ;\r
+ r2 : A /* l4 */\r
+ | (B)? => <<q>>? y /* l5 */\r
+ ;\r
+\r
+ In earlier versions the mrhoist routine would see that both l1 and\r
+ l2 contained predicates and would assume that this prevented either\r
+ from acting to suppress the other predicate. In the example above\r
+ it didn't realize the A at line l4 is capable of suppressing the\r
+ predicate at l1 even though alt l2 contains (indirectly) a predicate.\r
+\r
+ This is fixed in MR12b.\r
+\r
+ Reported by Reinier van den Born (reinier@vnet.ibm.com)\r
+\r
+#153. (Changed in MR12a) Bug in computation of -mrhoist suppression set\r
+\r
+ An oversight similar to that described in Item #152 appeared in\r
+ the computation of the set that "covered" a predicate. If a\r
+ predicate expression included a term such as p=AND(q,r) the context\r
+ of p was taken to be context(q) & context(r), when it should have\r
+ been context(q) | context(r). This is fixed in MR12a.\r
+\r
+#152. (Changed in MR12) Bug in generation of predicate expressions\r
+\r
+ The primary purpose for MR12 is to make quite clear that MR11 is\r
+ obsolete and to fix the bug related to predicate expressions.\r
+\r
+ In MR10 code was added to optimize the code generated for\r
+ predicate expression tests. Unfortunately, there was a\r
+ significant oversight in the code which resulted in a bug in\r
+ the generation of code for predicate expression tests which\r
+ contained predicates combined using AND:\r
+\r
+ r0 : (r1)* "@" ;\r
+ r1 : (AAA)? => <<p LATEXT(1)>>? r2 ;\r
+ r2 : (BBB)? => <<q LATEXT(1)>>? Q\r
+ | (BBB)? => <<r LATEXT(1)>>? Q\r
+ ;\r
+\r
+ In MR11 (and MR10 when using "-mrhoist on") the code generated\r
+ for r0 to predict r1 would be equivalent to:\r
+\r
+ if ( LA(1)==Q &&\r
+ (LA(1)==AAA && LA(1)==BBB) &&\r
+ ( p && ( q || r )) ) {\r
+\r
+ This is incorrect because it expresses the idea that LA(1)\r
+ *must* be AAA in order to attempt r1, and *must* be BBB to\r
+ attempt r2. The result was that r1 became unreachable since\r
+ both condition can not be simultaneously true.\r
+\r
+ The general philosophy of code generation for predicates\r
+ can be summarized as follows:\r
+\r
+ a. If the context is true don't enter an alt\r
+ for which the corresponding predicate is false.\r
+\r
+ If the context is false then it is okay to enter\r
+ the alt without evaluating the predicate at all.\r
+\r
+ b. A predicate created by ORing of predicates has\r
+ context which is the OR of their individual contexts.\r
+\r
+ c. A predicate created by ANDing of predicates has\r
+ (surprise) context which is the OR of their individual\r
+ contexts.\r
+\r
+ d. Apply these rules recursively.\r
+\r
+ e. Remember rule (a)\r
+\r
+ The correct code should express the idea that *if* LA(1) is\r
+ AAA then p must be true to attempt r1, but if LA(1) is *not*\r
+ AAA then it is okay to attempt r1, provided that *if* LA(1) is\r
+ BBB then one of q or r must be true.\r
+\r
+ if ( LA(1)==Q &&\r
+ ( !(LA(1)==AAA || LA(1)==BBB) ||\r
+ ( ! LA(1) == AAA || p) &&\r
+ ( ! LA(1) == BBB || q || r ) ) ) {\r
+\r
+ I believe this is fixed in MR12.\r
+\r
+ Reported by Reinier van den Born (reinier@vnet.ibm.com)\r
+\r
+#151a. (Changed in MR12) ANTLRParser::getLexer()\r
+\r
+ As a result of several requests, I have added public methods to\r
+ get a pointer to the lexer belonging to a parser.\r
+\r
+ ANTLRTokenStream *ANTLRParser::getLexer() const\r
+\r
+ Returns a pointer to the lexer being used by the\r
+ parser. ANTLRTokenStream is the base class of\r
+ DLGLexer\r
+\r
+ ANTLRTokenStream *ANTLRTokenBuffer::getLexer() const\r
+\r
+ Returns a pointer to the lexer being used by the\r
+ ANTLRTokenBuffer. ANTLRTokenStream is the base\r
+ class of DLGLexer\r
+\r
+ You must manually cast the ANTLRTokenStream to your program's\r
+ lexer class. Because the name of the lexer's class is not fixed.\r
+ Thus it is impossible to incorporate it into the DLGLexerBase\r
+ class.\r
+\r
+#151b.(Changed in MR12) ParserBlackBox member getLexer()\r
+\r
+ The template class ParserBlackBox now has a member getLexer()\r
+ which returns a pointer to the lexer.\r
+\r
+#150. (Changed in MR12) syntaxErrCount and lexErrCount now public\r
+\r
+ See Item #127 for more information.\r
+\r
+#149. (Changed in MR12) antlr option -info o (letter o for orphan)\r
+\r
+ If there is more than one rule which is not referenced by any\r
+ other rule then all such rules are listed. This is useful for\r
+ alerting one to rules which are not used, but which can still\r
+ contribute to ambiguity. For example:\r
+\r
+ start : a Z ;\r
+ unused: a A ;\r
+ a : (A)+ ;\r
+\r
+ will cause an ambiguity report for rule "a" which will be\r
+ difficult to understand if the user forgets about rule "unused"\r
+ simply because it is not used in the grammar.\r
+\r
+#148. (Changed in MR11) #token names appearing in zztokens,token_tbl\r
+\r
+ In a #token statement like the following:\r
+\r
+ #token Plus "\+"\r
+\r
+ the string "Plus" appears in the zztokens array (C mode) and\r
+ token_tbl (C++ mode). This string is used in most error\r
+ messages. In MR11 one has the option of using some other string,\r
+ (e.g. "+") in those tables.\r
+\r
+ In MR11 one can write:\r
+\r
+ #token Plus ("+") "\+"\r
+ #token RP ("(") "\("\r
+ #token COM ("comment begin") "/\*"\r
+\r
+ A #token statement is allowed to appear in more than one #lexclass\r
+ with different regular expressions. However, the token name appears\r
+ only once in the zztokens/token_tbl array. This means that only\r
+ one substitute can be specified for a given #token name. The second\r
+ attempt to define a substitute name (different from the first) will\r
+ result in an error message.\r
+\r
+#147. (Changed in MR11) Bug in follow set computation\r
+\r
+ There is a bug in 1.33 vanilla and all maintenance releases\r
+ prior to MR11 in the computation of the follow set. The bug is\r
+ different than that described in Item #82 and probably more\r
+ common. It was discovered in the ansi.g grammar while testing\r
+ the "ambiguity aid" (Item #119). The search for a bug started\r
+ when the ambiguity aid was unable to discover the actual source\r
+ of an ambiguity reported by antlr.\r
+\r
+ The problem appears when an optimization of the follow set\r
+ computation is used inappropriately. The result is that the\r
+ follow set used is the "worst case". In other words, the error\r
+ can lead to false reports of ambiguity. The good news is that\r
+ if you have a grammar in which you have addressed all reported\r
+ ambiguities you are ok. The bad news is that you may have spent\r
+ time fixing ambiguities that were not real, or used k=2 when\r
+ ck=2 might have been sufficient, and so on.\r
+\r
+ The following grammar demonstrates the problem:\r
+\r
+ ------------------------------------------------------------\r
+ expr : ID ;\r
+\r
+ start : stmt SEMI ;\r
+\r
+ stmt : CASE expr COLON\r
+ | expr SEMI\r
+ | plain_stmt\r
+ ;\r
+\r
+ plain_stmt : ID COLON ;\r
+ ------------------------------------------------------------\r
+\r
+ When compiled with k=1 and ck=2 it will report:\r
+\r
+ warning: alts 2 and 3 of the rule itself ambiguous upon\r
+ { IDENTIFIER }, { COLON }\r
+\r
+ When antlr analyzes "stmt" it computes the first[1] set of all\r
+ alternatives. It finds an ambiguity between alts 2 and 3 for ID.\r
+ It then computes the first[2] set for alternatives 2 and 3 to resolve\r
+ the ambiguity. In computing the first[2] set of "expr" (which is\r
+ only one token long) it needs to determine what could follow "expr".\r
+ Under a certain combination of circumstances antlr forgets that it\r
+ is trying to analyze "stmt" which can only be followed by SEMI and\r
+ adds to the first[2] set of "expr" the "global" follow set (including\r
+ "COLON") which could follow "expr" (under other conditions) in the\r
+ phrase "CASE expr COLON".\r
+\r
+#146. (Changed in MR11) Option -treport for locating "difficult" alts\r
+\r
+ It can be difficult to determine which alternatives are causing\r
+ pccts to work hard to resolve an ambiguity. In some cases the\r
+ ambiguity is successfully resolved after much CPU time so there\r
+ is no message at all.\r
+\r
+ A rough measure of the amount of work being peformed which is\r
+ independent of the CPU speed and system load is the number of\r
+ tnodes created. Using "-info t" gives information about the\r
+ total number of tnodes created and the peak number of tnodes.\r
+\r
+ Tree Nodes: peak 1300k created 1416k lost 0\r
+\r
+ It also puts in the generated C or C++ file the number of tnodes\r
+ created for a rule (at the end of the rule). However this\r
+ information is not sufficient to locate the alternatives within\r
+ a rule which are causing the creation of tnodes.\r
+\r
+ Using:\r
+\r
+ antlr -treport 100000 ....\r
+\r
+ causes antlr to list on stdout any alternatives which require the\r
+ creation of more than 100,000 tnodes, along with the lookahead sets\r
+ for those alternatives.\r
+\r
+ The following is a trivial case from the ansi.g grammar which shows\r
+ the format of the report. This report might be of more interest\r
+ in cases where 1,000,000 tuples were created to resolve the ambiguity.\r
+\r
+ -------------------------------------------------------------------------\r
+ There were 0 tuples whose ambiguity could not be resolved\r
+ by full lookahead\r
+ There were 157 tnodes created to resolve ambiguity between:\r
+\r
+ Choice 1: statement/2 line 475 file ansi.g\r
+ Choice 2: statement/3 line 476 file ansi.g\r
+\r
+ Intersection of lookahead[1] sets:\r
+\r
+ IDENTIFIER\r
+\r
+ Intersection of lookahead[2] sets:\r
+\r
+ LPARENTHESIS COLON AMPERSAND MINUS\r
+ STAR PLUSPLUS MINUSMINUS ONESCOMPLEMENT\r
+ NOT SIZEOF OCTALINT DECIMALINT\r
+ HEXADECIMALINT FLOATONE FLOATTWO IDENTIFIER\r
+ STRING CHARACTER\r
+ -------------------------------------------------------------------------\r
+\r
+#145. (Documentation) Generation of Expression Trees\r
+\r
+ Item #99 was misleading because it implied that the optimization\r
+ for tree expressions was available only for trees created by\r
+ predicate expressions and neglected to mention that it required\r
+ the use of "-mrhoist on". The optimization applies to tree\r
+ expressions created for grammars with k>1 and for predicates with\r
+ lookahead depth >1.\r
+\r
+ In MR11 the optimized version is always used so the -mrhoist on\r
+ option need not be specified.\r
+\r
+#144. (Changed in MR11) Incorrect test for exception group\r
+\r
+ In testing for a rule's exception group the label a pointer\r
+ is compared against '\0'. The intention is "*pointer".\r
+\r
+ Reported by Jeffrey C. Fried (Jeff@Fried.net).\r
+\r
+#143. (Changed in MR11) Optional ";" at end of #token statement\r
+\r
+ Fixes problem of:\r
+\r
+ #token X "x"\r
+\r
+ <<\r
+ parser action\r
+ >>\r
+\r
+ Being confused with:\r
+\r
+ #token X "x" <<lexical action>>\r
+\r
+#142. (Changed in MR11) class BufFileInput subclass of DLGInputStream\r
+\r
+ Alexey Demakov (demakov@kazbek.ispras.ru) has supplied class\r
+ BufFileInput derived from DLGInputStream which provides a\r
+ function lookahead(char *string) to test characters in the\r
+ input stream more than one character ahead.\r
+\r
+ The default amount of lookahead is specified by the constructor\r
+ and defaults to 8 characters. This does *not* include the one\r
+ character of lookahead maintained internally by DLG in member "ch"\r
+ and which is not available for testing via BufFileInput::lookahead().\r
+\r
+ This is a useful class for overcoming the one-character-lookahead\r
+ limitation of DLG without resorting to a lexer capable of\r
+ backtracking (like flex) which is not integrated with antlr as is\r
+ DLG.\r
+\r
+ There are no restrictions on copying or using BufFileInput.* except\r
+ that the authorship and related information must be retained in the\r
+ source code.\r
+\r
+ The class is located in pccts/h/BufFileInput.* of the kit.\r
+\r
+#141. (Changed in MR11) ZZDEBUG_CONSUME for ANTLRParser::consume()\r
+\r
+ A debug aid has been added to file ANTLRParser::consume() in\r
+ file AParser.cpp:\r
+\r
+ #ifdef ZZDEBUG_CONSUME_ACTION\r
+ zzdebug_consume_action();\r
+ #endif\r
+\r
+ Suggested by Sramji Ramanathan (ps@kumaran.com).\r
+\r
+#140. (Changed in MR11) #pred to define predicates\r
+\r
+ +---------------------------------------------------+\r
+ | Note: Assume "-prc on" for this entire discussion |\r
+ +---------------------------------------------------+\r
+\r
+ A problem with predicates is that each one is regarded as\r
+ unique and capable of disambiguating cases where two\r
+ alternatives have identical lookahead. For example:\r
+\r
+ rule : <<pred(LATEXT(1))>>? A\r
+ | <<pred(LATEXT(1))>>? A\r
+ ;\r
+\r
+ will not cause any error messages or warnings to be issued\r
+ by earlier versions of pccts. To compare the text of the\r
+ predicates is an incomplete solution.\r
+\r
+ In 1.33MR11 I am introducing the #pred statement in order to\r
+ solve some problems with predicates. The #pred statement allows\r
+ one to give a symbolic name to a "predicate literal" or a\r
+ "predicate expression" in order to refer to it in other predicate\r
+ expressions or in the rules of the grammar.\r
+\r
+ The predicate literal associated with a predicate symbol is C\r
+ or C++ code which can be used to test the condition. A\r
+ predicate expression defines a predicate symbol in terms of other\r
+ predicate symbols using "!", "&&", and "||". A predicate symbol\r
+ can be defined in terms of a predicate literal, a predicate\r
+ expression, or *both*.\r
+\r
+ When a predicate symbol is defined with both a predicate literal\r
+ and a predicate expression, the predicate literal is used to generate\r
+ code, but the predicate expression is used to check for two\r
+ alternatives with identical predicates in both alternatives.\r
+\r
+ Here are some examples of #pred statements:\r
+\r
+ #pred IsLabel <<isLabel(LATEXT(1))>>?\r
+ #pred IsLocalVar <<isLocalVar(LATEXT(1))>>?\r
+ #pred IsGlobalVar <<isGlobalVar(LATEXT(1)>>?\r
+ #pred IsVar <<isVar(LATEXT(1))>>? IsLocalVar || IsGlobalVar\r
+ #pred IsScoped <<isScoped(LATEXT(1))>>? IsLabel || IsLocalVar\r
+\r
+ I hope that the use of EBNF notation to describe the syntax of the\r
+ #pred statement will not cause problems for my readers (joke).\r
+\r
+ predStatement : "#pred"\r
+ CapitalizedName\r
+ (\r
+ "<<predicate_literal>>?"\r
+ | "<<predicate_literal>>?" predOrExpr\r
+ | predOrExpr\r
+ )\r
+ ;\r
+\r
+ predOrExpr : predAndExpr ( "||" predAndExpr ) * ;\r
+\r
+ predAndExpr : predPrimary ( "&&" predPrimary ) * ;\r
+\r
+ predPrimary : CapitalizedName\r
+ | "!" predPrimary\r
+ | "(" predOrExpr ")"\r
+ ;\r
+\r
+ What is the purpose of this nonsense ?\r
+\r
+ To understand how predicate symbols help, you need to realize that\r
+ predicate symbols are used in two different ways with two different\r
+ goals.\r
+\r
+ a. Allow simplification of predicates which have been combined\r
+ during predicate hoisting.\r
+\r
+ b. Allow recognition of identical predicates which can't disambiguate\r
+ alternatives with common lookahead.\r
+\r
+ First we will discuss goal (a). Consider the following rule:\r
+\r
+ rule0: rule1\r
+ | ID\r
+ | ...\r
+ ;\r
+\r
+ rule1: rule2\r
+ | rule3\r
+ ;\r
+\r
+ rule2: <<isX(LATEXT(1))>>? ID ;\r
+ rule3: <<!isX(LATEXT(1)>>? ID ;\r
+\r
+ When the predicates in rule2 and rule3 are combined by hoisting\r
+ to create a prediction expression for rule1 the result is:\r
+\r
+ if ( LA(1)==ID\r
+ && ( isX(LATEXT(1) || !isX(LATEXT(1) ) ) { rule1(); ...\r
+\r
+ This is inefficient, but more importantly, can lead to false\r
+ assumptions that the predicate expression distinguishes the rule1\r
+ alternative with some other alternative with lookahead ID. In\r
+ MR11 one can write:\r
+\r
+ #pred IsX <<isX(LATEXT(1))>>?\r
+\r
+ ...\r
+\r
+ rule2: <<IsX>>? ID ;\r
+ rule3: <<!IsX>>? ID ;\r
+\r
+ During hoisting MR11 recognizes this as a special case and\r
+ eliminates the predicates. The result is a prediction\r
+ expression like the following:\r
+\r
+ if ( LA(1)==ID ) { rule1(); ...\r
+\r
+ Please note that the following cases which appear to be equivalent\r
+ *cannot* be simplified by MR11 during hoisting because the hoisting\r
+ logic only checks for a "!" in the predicate action, not in the\r
+ predicate expression for a predicate symbol.\r
+\r
+ *Not* equivalent and is not simplified during hoisting:\r
+\r
+ #pred IsX <<isX(LATEXT(1))>>?\r
+ #pred NotX <<!isX(LATEXT(1))>>?\r
+ ...\r
+ rule2: <<IsX>>? ID ;\r
+ rule3: <<NotX>>? ID ;\r
+\r
+ *Not* equivalent and is not simplified during hoisting:\r
+\r
+ #pred IsX <<isX(LATEXT(1))>>?\r
+ #pred NotX !IsX\r
+ ...\r
+ rule2: <<IsX>>? ID ;\r
+ rule3: <<NotX>>? ID ;\r
+\r
+ Now we will discuss goal (b).\r
+\r
+ When antlr discovers that there is a lookahead ambiguity between\r
+ two alternatives it attempts to resolve the ambiguity by searching\r
+ for predicates in both alternatives. In the past any predicate\r
+ would do, even if the same one appeared in both alternatives:\r
+\r
+ rule: <<p(LATEXT(1))>>? X\r
+ | <<p(LATEXT(1))>>? X\r
+ ;\r
+\r
+ The #pred statement is a start towards solving this problem.\r
+ During ambiguity resolution (*not* predicate hoisting) the\r
+ predicates for the two alternatives are expanded and compared.\r
+ Consider the following example:\r
+\r
+ #pred Upper <<isUpper(LATEXT(1))>>?\r
+ #pred Lower <<isLower(LATEXT(1))>>?\r
+ #pred Alpha <<isAlpha(LATEXT(1))>>? Upper || Lower\r
+\r
+ rule0: rule1\r
+ | <<Alpha>>? ID\r
+ ;\r
+\r
+ rule1:\r
+ | rule2\r
+ | rule3\r
+ ...\r
+ ;\r
+\r
+ rule2: <<Upper>>? ID;\r
+ rule3: <<Lower>>? ID;\r
+\r
+ The definition of #pred Alpha expresses:\r
+\r
+ a. to test the predicate use the C code "isAlpha(LATEXT(1))"\r
+\r
+ b. to analyze the predicate use the information that\r
+ Alpha is equivalent to the union of Upper and Lower,\r
+\r
+ During ambiguity resolution the definition of Alpha is expanded\r
+ into "Upper || Lower" and compared with the predicate in the other\r
+ alternative, which is also "Upper || Lower". Because they are\r
+ identical MR11 will report a problem.\r
+\r
+ -------------------------------------------------------------------------\r
+ t10.g, line 5: warning: the predicates used to disambiguate rule rule0\r
+ (file t10.g alt 1 line 5 and alt 2 line 6)\r
+ are identical when compared without context and may have no\r
+ resolving power for some lookahead sequences.\r
+ -------------------------------------------------------------------------\r
+\r
+ If you use the "-info p" option the output file will contain:\r
+\r
+ +----------------------------------------------------------------------+\r
+ |#if 0 |\r
+ | |\r
+ |The following predicates are identical when compared without |\r
+ | lookahead context information. For some ambiguous lookahead |\r
+ | sequences they may not have any power to resolve the ambiguity. |\r
+ | |\r
+ |Choice 1: rule0/1 alt 1 line 5 file t10.g |\r
+ | |\r
+ | The original predicate for choice 1 with available context |\r
+ | information: |\r
+ | |\r
+ | OR expr |\r
+ | |\r
+ | pred << Upper>>? |\r
+ | depth=k=1 rule rule2 line 14 t10.g |\r
+ | set context: |\r
+ | ID |\r
+ | |\r
+ | pred << Lower>>? |\r
+ | depth=k=1 rule rule3 line 15 t10.g |\r
+ | set context: |\r
+ | ID |\r
+ | |\r
+ | The predicate for choice 1 after expansion (but without context |\r
+ | information): |\r
+ | |\r
+ | OR expr |\r
+ | |\r
+ | pred << isUpper(LATEXT(1))>>? |\r
+ | depth=k=1 rule line 1 t10.g |\r
+ | |\r
+ | pred << isLower(LATEXT(1))>>? |\r
+ | depth=k=1 rule line 2 t10.g |\r
+ | |\r
+ | |\r
+ |Choice 2: rule0/2 alt 2 line 6 file t10.g |\r
+ | |\r
+ | The original predicate for choice 2 with available context |\r
+ | information: |\r
+ | |\r
+ | pred << Alpha>>? |\r
+ | depth=k=1 rule rule0 line 6 t10.g |\r
+ | set context: |\r
+ | ID |\r
+ | |\r
+ | The predicate for choice 2 after expansion (but without context |\r
+ | information): |\r
+ | |\r
+ | OR expr |\r
+ | |\r
+ | pred << isUpper(LATEXT(1))>>? |\r
+ | depth=k=1 rule line 1 t10.g |\r
+ | |\r
+ | pred << isLower(LATEXT(1))>>? |\r
+ | depth=k=1 rule line 2 t10.g |\r
+ | |\r
+ | |\r
+ |#endif |\r
+ +----------------------------------------------------------------------+\r
+\r
+ The comparison of the predicates for the two alternatives takes\r
+ place without context information, which means that in some cases\r
+ the predicates will be considered identical even though they operate\r
+ on disjoint lookahead sets. Consider:\r
+\r
+ #pred Alpha\r
+\r
+ rule1: <<Alpha>>? ID\r
+ | <<Alpha>>? Label\r
+ ;\r
+\r
+ Because the comparison of predicates takes place without context\r
+ these will be considered identical. The reason for comparing\r
+ without context is that otherwise it would be necessary to re-evaluate\r
+ the entire predicate expression for each possible lookahead sequence.\r
+ This would require more code to be written and more CPU time during\r
+ grammar analysis, and it is not yet clear whether anyone will even make\r
+ use of the new #pred facility.\r
+\r
+ A temporary workaround might be to use different #pred statements\r
+ for predicates you know have different context. This would avoid\r
+ extraneous warnings.\r
+\r
+ The above example might be termed a "false positive". Comparison\r
+ without context will also lead to "false negatives". Consider the\r
+ following example:\r
+\r
+ #pred Alpha\r
+ #pred Beta\r
+\r
+ rule1: <<Alpha>>? A\r
+ | rule2\r
+ ;\r
+\r
+ rule2: <<Alpha>>? A\r
+ | <<Beta>>? B\r
+ ;\r
+\r
+ The predicate used for alt 2 of rule1 is (Alpha || Beta). This\r
+ appears to be different than the predicate Alpha used for alt1.\r
+ However, the context of Beta is B. Thus when the lookahead is A\r
+ Beta will have no resolving power and Alpha will be used for both\r
+ alternatives. Using the same predicate for both alternatives isn't\r
+ very helpful, but this will not be detected with 1.33MR11.\r
+\r
+ To properly handle this the predicate expression would have to be\r
+ evaluated for each distinct lookahead context.\r
+\r
+ To determine whether two predicate expressions are identical is\r
+ difficult. The routine may fail to identify identical predicates.\r
+\r
+ The #pred feature also compares predicates to see if a choice between\r
+ alternatives which is resolved by a predicate which makes the second\r
+ choice unreachable. Consider the following example:\r
+\r
+ #pred A <<A(LATEXT(1)>>?\r
+ #pred B <<B(LATEXT(1)>>?\r
+ #pred A_or_B A || B\r
+\r
+ r : s\r
+ | t\r
+ ;\r
+ s : <<A_or_B>>? ID\r
+ ;\r
+ t : <<A>>? ID\r
+ ;\r
+\r
+ ----------------------------------------------------------------------------\r
+ t11.g, line 5: warning: the predicate used to disambiguate the\r
+ first choice of rule r\r
+ (file t11.g alt 1 line 5 and alt 2 line 6)\r
+ appears to "cover" the second predicate when compared without context.\r
+ The second predicate may have no resolving power for some lookahead\r
+ sequences.\r
+ ----------------------------------------------------------------------------\r
+\r
+#139. (Changed in MR11) Problem with -gp in C++ mode\r
+\r
+ The -gp option to add a prefix to rule names did not work in\r
+ C++ mode. This has been fixed.\r
+\r
+ Reported by Alexey Demakov (demakov@kazbek.ispras.ru).\r
+\r
+#138. (Changed in MR11) Additional makefiles for non-MSVC++ MS systems\r
+\r
+ Sramji Ramanathan (ps@kumaran.com) has supplied makefiles for\r
+ building antlr and dlg with Win95/NT development tools that\r
+ are not based on MSVC5. They are pccts/antlr/AntlrMS.mak and\r
+ pccts/dlg/DlgMS.mak.\r
+\r
+ The first line of the makefiles require a definition of PCCTS_HOME.\r
+\r
+ These are in additiion to the AntlrMSVC50.* and DlgMSVC50.*\r
+ supplied by Jeff Vincent (JVincent@novell.com).\r
+\r
+#137. (Changed in MR11) Token getType(), getText(), getLine() const members\r
+\r
+ --------------------------------------------------------------------\r
+ If you use ANTLRCommonToken this change probably does not affect you.\r
+ --------------------------------------------------------------------\r
+\r
+ For a long time it has bothered me that these accessor functions\r
+ in ANTLRAbstractToken were not const member functions. I have\r
+ refrained from changing them because it require users to modify\r
+ existing token class definitions which are derived directly\r
+ from ANTLRAbstractToken. I think it is now time.\r
+\r
+ For those who are not used to C++, a "const member function" is a\r
+ member function which does not modify its own object - the thing\r
+ to which "this" points. This is quite different from a function\r
+ which does not modify its arguments\r
+\r
+ Most token definitions based on ANTLRAbstractToken have something like\r
+ the following in order to create concrete definitions of the pure\r
+ virtual methods in ANTLRAbstractToken:\r
+\r
+ class MyToken : public ANTLRAbstractToken {\r
+ ...\r
+ ANTLRTokenType getType() {return _type; }\r
+ int getLine() {return _line; }\r
+ ANTLRChar * getText() {return _text; }\r
+ ...\r
+ }\r
+\r
+ The required change is simply to put "const" following the function\r
+ prototype in the header (.h file) and the definition file (.cpp if\r
+ it is not inline):\r
+\r
+ class MyToken : public ANTLRAbstractToken {\r
+ ...\r
+ ANTLRTokenType getType() const {return _type; }\r
+ int getLine() const {return _line; }\r
+ ANTLRChar * getText() const {return _text; }\r
+ ...\r
+ }\r
+\r
+ This was originally proposed a long time ago by Bruce\r
+ Guenter (bruceg@qcc.sk.ca).\r
+\r
+#136. (Changed in MR11) Added getLength() to ANTLRCommonToken\r
+\r
+ Classes ANTLRCommonToken and ANTLRCommonTokenNoRefCountToken\r
+ now have a member function:\r
+\r
+ int getLength() const { return strlen(getText()) }\r
+\r
+ Suggested by Sramji Ramanathan (ps@kumaran.com).\r
+\r
+#135. (Changed in MR11) Raised antlr's own default ZZLEXBUFSIZE to 8k\r
+\r
+#134a. (ansi_mr10.zip) T.J. Parr's ANSI C grammar made 1.33MR11 compatible\r
+\r
+ There is a typographical error in the definition of BITWISEOREQ:\r
+\r
+ #token BITWISEOREQ "!=" should be "\|="\r
+\r
+ When this change is combined with the bugfix to the follow set cache\r
+ problem (Item #147) and a minor rearrangement of the grammar\r
+ (Item #134b) it becomes a k=1 ck=2 grammar.\r
+\r
+#134b. (ansi_mr10.zip) T.J. Parr's ANSI C grammar made 1.33MR11 compatible\r
+\r
+ The following changes were made in the ansi.g grammar (along with\r
+ using -mrhoist on):\r
+\r
+ ansi.g\r
+ ======\r
+ void tracein(char *) ====> void tracein(const char *)\r
+ void traceout(char *) ====> void traceout(const char *)\r
+\r
+ <LT(1)->getType()==IDENTIFIER ? isTypeName(LT(1)->getText()) : 1>>?\r
+ ====> <<isTypeName(LT(1)->getText())>>?\r
+\r
+ <<(LT(1)->getType()==LPARENTHESIS && LT(2)->getType()==IDENTIFIER) ? \\r
+ isTypeName(LT(2)->getText()) : 1>>?\r
+ ====> (LPARENTHESIS IDENTIFIER)? => <<isTypeName(LT(2)->getText())>>?\r
+\r
+ <<(LT(1)->getType()==LPARENTHESIS && LT(2)->getType()==IDENTIFIER) ? \\r
+ isTypeName(LT(2)->getText()) : 1>>?\r
+ ====> (LPARENTHESIS IDENTIFIER)? => <<isTypeName(LT(2)->getText())>>?\r
+\r
+ added to init(): traceOptionValueDefault=0;\r
+ added to init(): traceOption(-1);\r
+\r
+ change rule "statement":\r
+\r
+ statement\r
+ : plain_label_statement\r
+ | case_label_statement\r
+ | <<;>> expression SEMICOLON\r
+ | compound_statement\r
+ | selection_statement\r
+ | iteration_statement\r
+ | jump_statement\r
+ | SEMICOLON\r
+ ;\r
+\r
+ plain_label_statement\r
+ : IDENTIFIER COLON statement\r
+ ;\r
+\r
+ case_label_statement\r
+ : CASE constant_expression COLON statement\r
+ | DEFAULT COLON statement\r
+ ;\r
+\r
+ support.cpp\r
+ ===========\r
+ void tracein(char *) ====> void tracein(const char *)\r
+ void traceout(char *) ====> void traceout(const char *)\r
+\r
+ added to tracein(): ANTLRParser::tracein(r); // call superclass method\r
+ added to traceout(): ANTLRParser::traceout(r); // call superclass method\r
+\r
+ Makefile\r
+ ========\r
+ added to AFLAGS: -mrhoist on -prc on\r
+\r
+#133. (Changed in 1.33MR11) Make trace options public in ANTLRParser\r
+\r
+ In checking T.J. Parr's ANSI C grammar for compatibility with\r
+ 1.33MR11 discovered that it was inconvenient to have the\r
+ trace facilities with protected access.\r
+\r
+#132. (Changed in 1.33MR11) Recognition of identical predicates in alts\r
+\r
+ Prior to 1.33MR11, there would be no ambiguity warning when the\r
+ very same predicate was used to disambiguate both alternatives:\r
+\r
+ test: ref B\r
+ | ref C\r
+ ;\r
+\r
+ ref : <<pred(LATEXT(1)>>? A\r
+\r
+ In 1.33MR11 this will cause the warning:\r
+\r
+ warning: the predicates used to disambiguate rule test\r
+ (file v98.g alt 1 line 1 and alt 2 line 2)\r
+ are identical and have no resolving power\r
+\r
+ ----------------- Note -----------------\r
+\r
+ This is different than the following case\r
+\r
+ test: <<pred(LATEXT(1))>>? A B\r
+ | <<pred(LATEXT(1)>>? A C\r
+ ;\r
+\r
+ In this case there are two distinct predicates\r
+ which have exactly the same text. In the first\r
+ example there are two references to the same\r
+ predicate. The problem represented by this\r
+ grammar will be addressed later.\r
+\r
+#131. (Changed in 1.33MR11) Case insensitive command line options\r
+\r
+ Command line switches like "-CC" and keywords like "on", "off",\r
+ and "stdin" are no longer case sensitive in antlr, dlg, and sorcerer.\r
+\r
+#130. (Changed in 1.33MR11) Changed ANTLR_VERSION to int from string\r
+\r
+ The ANTLR_VERSION was not an integer, making it difficult to\r
+ perform conditional compilation based on the antlr version.\r
+\r
+ Henceforth, ANTLR_VERSION will be:\r
+\r
+ (base_version * 10000) + release number\r
+\r
+ thus 1.33MR11 will be: 133*100+11 = 13311\r
+\r
+ Suggested by Rainer Janssen (Rainer.Janssen@Informatik.Uni-Oldenburg.DE).\r
+\r
+#129. (Changed in 1.33MR11) Addition of ANTLR_VERSION to <parserName>.h\r
+\r
+ The following code is now inserted into <parserName>.h amd\r
+ stdpccts.h:\r
+\r
+ #ifndef ANTLR_VERSION\r
+ #define ANTLR_VERSION 13311\r
+ #endif\r
+\r
+ Suggested by Rainer Janssen (Rainer.Janssen@Informatik.Uni-Oldenburg.DE)\r
+\r
+#128. (Changed in 1.33MR11) Redundant predicate code in (<<pred>>? ...)+\r
+\r
+ Prior to 1.33MR11, the following grammar would generate\r
+ redundant tests for the "while" condition.\r
+\r
+ rule2 : (<<pred>>? X)+ X\r
+ | B\r
+ ;\r
+\r
+ The code would resemble:\r
+\r
+ if (LA(1)==X) {\r
+ if (pred) {\r
+ do {\r
+ if (!pred) {zzfailed_pred(" pred");}\r
+ zzmatch(X); zzCONSUME;\r
+ } while (LA(1)==X && pred && pred);\r
+ } else {...\r
+\r
+ With 1.33MR11 the redundant predicate test is omitted.\r
+\r
+#127. (Changed in 1.33MR11)\r
+\r
+ Count Syntax Errors Count DLG Errors\r
+ ------------------- ----------------\r
+\r
+ C++ mode ANTLRParser:: DLGLexerBase::\r
+ syntaxErrCount lexErrCount\r
+ C mode zzSyntaxErrCount zzLexErrCount\r
+\r
+ The C mode variables are global and initialized to 0.\r
+ They are *not* reset to 0 automatically when antlr is\r
+ restarted.\r
+\r
+ The C++ mode variables are public. They are initialized\r
+ to 0 by the constructors. They are *not* reset to 0 by the\r
+ ANTLRParser::init() method.\r
+\r
+ Suggested by Reinier van den Born (reinier@vnet.ibm.com).\r
+\r
+#126. (Changed in 1.33MR11) Addition of #first <<...>>\r
+\r
+ The #first <<...>> inserts the specified text in the output\r
+ files before any other #include statements required by pccts.\r
+ The only things before the #first text are comments and\r
+ a #define ANTLR_VERSION.\r
+\r
+ Requested by and Esa Pulkkinen (esap@cs.tut.fi) and Alexin\r
+ Zoltan (alexin@inf.u-szeged.hu).\r
+\r
+#125. (Changed in 1.33MR11) Lookahead for (guard)? && <<p>>? predicates\r
+\r
+ When implementing the new style of guard predicate (Item #113)\r
+ in 1.33MR10 I decided to temporarily ignore the problem of\r
+ computing the "narrowest" lookahead context.\r
+\r
+ Consider the following k=1 grammar:\r
+\r
+ start : a\r
+ | b\r
+ ;\r
+\r
+ a : (A)? && <<pred1(LATEXT(1))>>? ab ;\r
+ b : (B)? && <<pred2(LATEXT(1))>>? ab ;\r
+\r
+ ab : A | B ;\r
+\r
+ In MR10 the context for both "a" and "b" was {A B} because this is\r
+ the first set of rule "ab". Normally, this is not a problem because\r
+ the predicate which follows the guard inhibits any ambiguity report\r
+ by antlr.\r
+\r
+ In MR11 the first set for rule "a" is {A} and for rule "b" it is {B}.\r
+\r
+#124. A Note on the New "&&" Style Guarded Predicates\r
+\r
+ I've been asked several times, "What is the difference between\r
+ the old "=>" style guard predicates and the new style "&&" guard\r
+ predicates, and how do you choose one over the other" ?\r
+\r
+ The main difference is that the "=>" does not apply the\r
+ predicate if the context guard doesn't match, whereas\r
+ the && form always does. What is the significance ?\r
+\r
+ If you have a predicate which is not on the "leading edge"\r
+ it is cannot be hoisted. Suppose you need a predicate that\r
+ looks at LA(2). You must introduce it manually. The\r
+ classic example is:\r
+\r
+ castExpr :\r
+ LP typeName RP\r
+ | ....\r
+ ;\r
+\r
+ typeName : <<isTypeName(LATEXT(1))>>? ID\r
+ | STRUCT ID\r
+ ;\r
+\r
+ The problem is that isTypeName() isn't on the leading edge\r
+ of typeName, so it won't be hoisted into castExpr to help\r
+ make a decision on which production to choose.\r
+\r
+ The *first* attempt to fix it is this:\r
+\r
+ castExpr :\r
+ <<isTypeName(LATEXT(2))>>?\r
+ LP typeName RP\r
+ | ....\r
+ ;\r
+\r
+ Unfortunately, this won't work because it ignores\r
+ the problem of STRUCT. The solution is to apply\r
+ isTypeName() in castExpr if LA(2) is an ID and\r
+ don't apply it when LA(2) is STRUCT:\r
+\r
+ castExpr :\r
+ (LP ID)? => <<isTypeName(LATEXT(2))>>?\r
+ LP typeName RP\r
+ | ....\r
+ ;\r
+\r
+ In conclusion, the "=>" style guarded predicate is\r
+ useful when:\r
+\r
+ a. the tokens required for the predicate\r
+ are not on the leading edge\r
+ b. there are alternatives in the expression\r
+ selected by the predicate for which the\r
+ predicate is inappropriate\r
+\r
+ If (b) were false, then one could use a simple\r
+ predicate (assuming "-prc on"):\r
+\r
+ castExpr :\r
+ <<isTypeName(LATEXT(2))>>?\r
+ LP typeName RP\r
+ | ....\r
+ ;\r
+\r
+ typeName : <<isTypeName(LATEXT(1))>>? ID\r
+ ;\r
+\r
+ So, when do you use the "&&" style guarded predicate ?\r
+\r
+ The new-style "&&" predicate should always be used with\r
+ predicate context. The context guard is in ADDITION to\r
+ the automatically computed context. Thus it useful for\r
+ predicates which depend on the token type for reasons\r
+ other than context.\r
+\r
+ The following example is contributed by Reinier van den Born\r
+ (reinier@vnet.ibm.com).\r
+\r
+ +-------------------------------------------------------------------------+\r
+ | This grammar has two ways to call functions: |\r
+ | |\r
+ | - a "standard" call syntax with parens and comma separated args |\r
+ | - a shell command like syntax (no parens and spacing separated args) |\r
+ | |\r
+ | The former also allows a variable to hold the name of the function, |\r
+ | the latter can also be used to call external commands. |\r
+ | |\r
+ | The grammar (simplified) looks like this: |\r
+ | |\r
+ | fun_call : ID "(" { expr ("," expr)* } ")" |\r
+ | /* ID is function name */ |\r
+ | | "@" ID "(" { expr ("," expr)* } ")" |\r
+ | /* ID is var containing fun name */ |\r
+ | ; |\r
+ | |\r
+ | command : ID expr* /* ID is function name */ |\r
+ | | path expr* /* path is external command name */ |\r
+ | ; |\r
+ | |\r
+ | path : ID /* left out slashes and such */ |\r
+ | | "@" ID /* ID is environment var */ |\r
+ | ; |\r
+ | |\r
+ | expr : .... |\r
+ | | "(" expr ")"; |\r
+ | |\r
+ | call : fun_call |\r
+ | | command |\r
+ | ; |\r
+ | |\r
+ | Obviously the call is wildly ambiguous. This is more or less how this |\r
+ | is to be resolved: |\r
+ | |\r
+ | A call begins with an ID or an @ followed by an ID. |\r
+ | |\r
+ | If it is an ID and if it is an ext. command name -> command |\r
+ | if followed by a paren -> fun_call |\r
+ | otherwise -> command |\r
+ | |\r
+ | If it is an @ and if the ID is a var name -> fun_call |\r
+ | otherwise -> command |\r
+ | |\r
+ | One can implement these rules quite neatly using && predicates: |\r
+ | |\r
+ | call : ("@" ID)? && <<isVarName(LT(2))>>? fun_call |\r
+ | | (ID)? && <<isExtCmdName>>? command |\r
+ | | (ID "(")? fun_call |\r
+ | | command |\r
+ | ; |\r
+ | |\r
+ | This can be done better, so it is not an ideal example, but it |\r
+ | conveys the principle. |\r
+ +-------------------------------------------------------------------------+\r
+\r
+#123. (Changed in 1.33MR11) Correct definition of operators in ATokPtr.h\r
+\r
+ The return value of operators in ANTLRTokenPtr:\r
+\r
+ changed: unsigned ... operator !=(...)\r
+ to: int ... operator != (...)\r
+ changed: unsigned ... operator ==(...)\r
+ to: int ... operator == (...)\r
+\r
+ Suggested by R.A. Nelson (cowboy@VNET.IBM.COM)\r
+\r
+#122. (Changed in 1.33MR11) Member functions to reset DLG in C++ mode\r
+\r
+ void DLGFileReset(FILE *f) { input = f; found_eof = 0; }\r
+ void DLGStringReset(DLGChar *s) { input = s; p = &input[0]; }\r
+\r
+ Supplied by R.A. Nelson (cowboy@VNET.IBM.COM)\r
+\r
+#121. (Changed in 1.33MR11) Another attempt to fix -o (output dir) option\r
+\r
+ Another attempt is made to improve the -o option of antlr, dlg,\r
+ and sorcerer. This one by JVincent (JVincent@novell.com).\r
+\r
+ The current rule:\r
+\r
+ a. If -o is not specified than any explicit directory\r
+ names are retained.\r
+\r
+ b. If -o is specified than the -o directory name overrides any\r
+ explicit directory names.\r
+\r
+ c. The directory name of the grammar file is *not* stripped\r
+ to create the main output file. However it is stil subject\r
+ to override by the -o directory name.\r
+\r
+#120. (Changed in 1.33MR11) "-info f" output to stdout rather than stderr\r
+\r
+ Added option 0 (e.g. "-info 0") which is a noop.\r
+\r
+#119. (Changed in 1.33MR11) Ambiguity aid for grammars\r
+\r
+ The user can ask for additional information on ambiguities reported\r
+ by antlr to stdout. At the moment, only one ambiguity report can\r
+ be created in an antlr run.\r
+\r
+ This feature is enabled using the "-aa" (Ambiguity Aid) option.\r
+\r
+ The following options control the reporting of ambiguities:\r
+\r
+ -aa ruleName Selects reporting by name of rule\r
+ -aa lineNumber Selects reporting by line number\r
+ (file name not compared)\r
+\r
+ -aam Selects "multiple" reporting for a token\r
+ in the intersection set of the\r
+ alternatives.\r
+\r
+ For instance, the token ID may appear dozens\r
+ of times in various paths as the program\r
+ explores the rules which are reachable from\r
+ the point of an ambiguity. With option -aam\r
+ every possible path the search program\r
+ encounters is reported.\r
+\r
+ Without -aam only the first encounter is\r
+ reported. This may result in incomplete\r
+ information, but the information may be\r
+ sufficient and much shorter.\r
+\r
+ -aad depth Selects the depth of the search.\r
+ The default value is 1.\r
+\r
+ The number of paths to be searched, and the\r
+ size of the report can grow geometrically\r
+ with the -ck value if a full search for all\r
+ contributions to the source of the ambiguity\r
+ is explored.\r
+\r
+ The depth represents the number of tokens\r
+ in the lookahead set which are matched against\r
+ the set of ambiguous tokens. A depth of 1\r
+ means that the search stops when a lookahead\r
+ sequence of just one token is matched.\r
+\r
+ A k=1 ck=6 grammar might generate 5,000 items\r
+ in a report if a full depth 6 search is made\r
+ with the Ambiguity Aid. The source of the\r
+ problem may be in the first token and obscured\r
+ by the volume of data - I hesitate to call\r
+ it information.\r
+\r
+ When the user selects a depth > 1, the search\r
+ is first performed at depth=1 for both\r
+ alternatives, then depth=2 for both alternatives,\r
+ etc.\r
+\r
+ Sample output for rule grammar in antlr.g itself:\r
+\r
+ +---------------------------------------------------------------------+\r
+ | Ambiguity Aid |\r
+ | |\r
+ | Choice 1: grammar/70 line 632 file a.g |\r
+ | Choice 2: grammar/82 line 644 file a.g |\r
+ | |\r
+ | Intersection of lookahead[1] sets: |\r
+ | |\r
+ | "\}" "class" "#errclass" "#tokclass" |\r
+ | |\r
+ | Choice:1 Depth:1 Group:1 ("#errclass") |\r
+ | 1 in (...)* block grammar/70 line 632 a.g |\r
+ | 2 to error grammar/73 line 635 a.g |\r
+ | 3 error error/1 line 894 a.g |\r
+ | 4 #token "#errclass" error/2 line 895 a.g |\r
+ | |\r
+ | Choice:1 Depth:1 Group:2 ("#tokclass") |\r
+ | 2 to tclass grammar/74 line 636 a.g |\r
+ | 3 tclass tclass/1 line 937 a.g |\r
+ | 4 #token "#tokclass" tclass/2 line 938 a.g |\r
+ | |\r
+ | Choice:1 Depth:1 Group:3 ("class") |\r
+ | 2 to class_def grammar/75 line 637 a.g |\r
+ | 3 class_def class_def/1 line 669 a.g |\r
+ | 4 #token "class" class_def/3 line 671 a.g |\r
+ | |\r
+ | Choice:1 Depth:1 Group:4 ("\}") |\r
+ | 2 #token "\}" grammar/76 line 638 a.g |\r
+ | |\r
+ | Choice:2 Depth:1 Group:5 ("#errclass") |\r
+ | 1 in (...)* block grammar/83 line 645 a.g |\r
+ | 2 to error grammar/93 line 655 a.g |\r
+ | 3 error error/1 line 894 a.g |\r
+ | 4 #token "#errclass" error/2 line 895 a.g |\r
+ | |\r
+ | Choice:2 Depth:1 Group:6 ("#tokclass") |\r
+ | 2 to tclass grammar/94 line 656 a.g |\r
+ | 3 tclass tclass/1 line 937 a.g |\r
+ | 4 #token "#tokclass" tclass/2 line 938 a.g |\r
+ | |\r
+ | Choice:2 Depth:1 Group:7 ("class") |\r
+ | 2 to class_def grammar/95 line 657 a.g |\r
+ | 3 class_def class_def/1 line 669 a.g |\r
+ | 4 #token "class" class_def/3 line 671 a.g |\r
+ | |\r
+ | Choice:2 Depth:1 Group:8 ("\}") |\r
+ | 2 #token "\}" grammar/96 line 658 a.g |\r
+ +---------------------------------------------------------------------+\r
+\r
+ For a linear lookahead set ambiguity (where k=1 or for k>1 but\r
+ when all lookahead sets [i] with i<k all have degree one) the\r
+ reports appear in the following order:\r
+\r
+ for (depth=1 ; depth <= "-aad depth" ; depth++) {\r
+ for (alternative=1; alternative <=2 ; alternative++) {\r
+ while (matches-are-found) {\r
+ group++;\r
+ print-report\r
+ };\r
+ };\r
+ };\r
+\r
+ For reporting a k-tuple ambiguity, the reports appear in the\r
+ following order:\r
+\r
+ for (depth=1 ; depth <= "-aad depth" ; depth++) {\r
+ while (matches-are-found) {\r
+ for (alternative=1; alternative <=2 ; alternative++) {\r
+ group++;\r
+ print-report\r
+ };\r
+ };\r
+ };\r
+\r
+ This is because matches are generated in different ways for\r
+ linear lookahead and k-tuples.\r
+\r
+#118. (Changed in 1.33MR11) DEC VMS makefile and VMS related changes\r
+\r
+ Revised makefiles for DEC/VMS operating system for antlr, dlg,\r
+ and sorcerer.\r
+\r
+ Reduced names of routines with external linkage to less than 32\r
+ characters to conform to DEC/VMS linker limitations.\r
+\r
+ Jean-Francois Pieronne discovered problems with dlg and antlr\r
+ due to the VMS linker not being case sensitive for names with\r
+ external linkage. In dlg the problem was with "className" and\r
+ "ClassName". In antlr the problem was with "GenExprSets" and\r
+ "genExprSets".\r
+\r
+ Added genmms, a version of genmk for the DEC/VMS version of make.\r
+ The source is in directory pccts/support/DECmms.\r
+\r
+ All VMS contributions by Jean-Francois Pieronne (jfp@iname.com).\r
+\r
+#117. (Changed in 1.33MR10) new EXPERIMENTAL predicate hoisting code\r
+\r
+ The hoisting of predicates into rules to create prediction\r
+ expressions is a problem in antlr. Consider the following\r
+ example (k=1 with -prc on):\r
+\r
+ start : (a)* "@" ;\r
+ a : b | c ;\r
+ b : <<isUpper(LATEXT(1))>>? A ;\r
+ c : A ;\r
+\r
+ Prior to 1.33MR10 the code generated for "start" would resemble:\r
+\r
+ while {\r
+ if (LA(1)==A &&\r
+ (!LA(1)==A || isUpper())) {\r
+ a();\r
+ }\r
+ };\r
+\r
+ This code is wrong because it makes rule "c" unreachable from\r
+ "start". The essence of the problem is that antlr fails to\r
+ recognize that there can be a valid alternative within "a" even\r
+ when the predicate <<isUpper(LATEXT(1))>>? is false.\r
+\r
+ In 1.33MR10 with -mrhoist the hoisting of the predicate into\r
+ "start" is suppressed because it recognizes that "c" can\r
+ cover all the cases where the predicate is false:\r
+\r
+ while {\r
+ if (LA(1)==A) {\r
+ a();\r
+ }\r
+ };\r
+\r
+ With the antlr "-info p" switch the user will receive information\r
+ about the predicate suppression in the generated file:\r
+\r
+ --------------------------------------------------------------\r
+ #if 0\r
+\r
+ Hoisting of predicate suppressed by alternative without predicate.\r
+ The alt without the predicate includes all cases where\r
+ the predicate is false.\r
+\r
+ WITH predicate: line 7 v1.g\r
+ WITHOUT predicate: line 7 v1.g\r
+\r
+ The context set for the predicate:\r
+\r
+ A\r
+\r
+ The lookahead set for the alt WITHOUT the semantic predicate:\r
+\r
+ A\r
+\r
+ The predicate:\r
+\r
+ pred << isUpper(LATEXT(1))>>?\r
+ depth=k=1 rule b line 9 v1.g\r
+ set context:\r
+ A\r
+ tree context: null\r
+\r
+ Chain of referenced rules:\r
+\r
+ #0 in rule start (line 5 v1.g) to rule a\r
+ #1 in rule a (line 7 v1.g)\r
+\r
+ #endif\r
+ --------------------------------------------------------------\r
+\r
+ A predicate can be suppressed by a combination of alternatives\r
+ which, taken together, cover a predicate:\r
+\r
+ start : (a)* "@" ;\r
+\r
+ a : b | ca | cb | cc ;\r
+\r
+ b : <<isUpper(LATEXT(1))>>? ( A | B | C ) ;\r
+\r
+ ca : A ;\r
+ cb : B ;\r
+ cc : C ;\r
+\r
+ Consider a more complex example in which "c" covers only part of\r
+ a predicate:\r
+\r
+ start : (a)* "@" ;\r
+\r
+ a : b\r
+ | c\r
+ ;\r
+\r
+ b : <<isUpper(LATEXT(1))>>?\r
+ ( A\r
+ | X\r
+ );\r
+\r
+ c : A\r
+ ;\r
+\r
+ Prior to 1.33MR10 the code generated for "start" would resemble:\r
+\r
+ while {\r
+ if ( (LA(1)==A || LA(1)==X) &&\r
+ (! (LA(1)==A || LA(1)==X) || isUpper()) {\r
+ a();\r
+ }\r
+ };\r
+\r
+ With 1.33MR10 and -mrhoist the predicate context is restricted to\r
+ the non-covered lookahead. The code resembles:\r
+\r
+ while {\r
+ if ( (LA(1)==A || LA(1)==X) &&\r
+ (! (LA(1)==X) || isUpper()) {\r
+ a();\r
+ }\r
+ };\r
+\r
+ With the antlr "-info p" switch the user will receive information\r
+ about the predicate restriction in the generated file:\r
+\r
+ --------------------------------------------------------------\r
+ #if 0\r
+\r
+ Restricting the context of a predicate because of overlap\r
+ in the lookahead set between the alternative with the\r
+ semantic predicate and one without\r
+ Without this restriction the alternative without the predicate\r
+ could not be reached when input matched the context of the\r
+ predicate and the predicate was false.\r
+\r
+ WITH predicate: line 11 v4.g\r
+ WITHOUT predicate: line 12 v4.g\r
+\r
+ The original context set for the predicate:\r
+\r
+ A X\r
+\r
+ The lookahead set for the alt WITHOUT the semantic predicate:\r
+\r
+ A\r
+\r
+ The intersection of the two sets\r
+\r
+ A\r
+\r
+ The original predicate:\r
+\r
+ pred << isUpper(LATEXT(1))>>?\r
+ depth=k=1 rule b line 15 v4.g\r
+ set context:\r
+ A X\r
+ tree context: null\r
+\r
+ The new (modified) form of the predicate:\r
+\r
+ pred << isUpper(LATEXT(1))>>?\r
+ depth=k=1 rule b line 15 v4.g\r
+ set context:\r
+ X\r
+ tree context: null\r
+\r
+ #endif\r
+ --------------------------------------------------------------\r
+\r
+ The bad news about -mrhoist:\r
+\r
+ (a) -mrhoist does not analyze predicates with lookahead\r
+ depth > 1.\r
+\r
+ (b) -mrhoist does not look past a guarded predicate to\r
+ find context which might cover other predicates.\r
+\r
+ For these cases you might want to use syntactic predicates.\r
+ When a semantic predicate fails during guess mode the guess\r
+ fails and the next alternative is tried.\r
+\r
+ Limitation (a) is illustrated by the following example:\r
+\r
+ start : (stmt)* EOF ;\r
+\r
+ stmt : cast\r
+ | expr\r
+ ;\r
+ cast : <<isTypename(LATEXT(2))>>? LP ID RP ;\r
+\r
+ expr : LP ID RP ;\r
+\r
+ This is not much different from the first example, except that\r
+ it requires two tokens of lookahead context to determine what\r
+ to do. This predicate is NOT suppressed because the current version\r
+ is unable to handle predicates with depth > 1.\r
+\r
+ A predicate can be combined with other predicates during hoisting.\r
+ In those cases the depth=1 predicates are still handled. Thus,\r
+ in the following example the isUpper() predicate will be suppressed\r
+ by line #4 when hoisted from "bizarre" into "start", but will still\r
+ be present in "bizarre" in order to predict "stmt".\r
+\r
+ start : (bizarre)* EOF ; // #1\r
+ // #2\r
+ bizarre : stmt // #3\r
+ | A // #4\r
+ ;\r
+\r
+ stmt : cast\r
+ | expr\r
+ ;\r
+\r
+ cast : <<isTypename(LATEXT(2))>>? LP ID RP ;\r
+\r
+ expr : LP ID RP ;\r
+ | <<isUpper(LATEXT(1))>>? A\r
+\r
+ Limitation (b) is illustrated by the following example of a\r
+ context guarded predicate:\r
+\r
+ rule : (A)? <<p>>? // #1\r
+ (A // #2\r
+ |B // #3\r
+ ) // #4\r
+ | <<q>> B // #5\r
+ ;\r
+\r
+ Recall that this means that when the lookahead is NOT A then\r
+ the predicate "p" is ignored and it attempts to match "A|B".\r
+ Ideally, the "B" at line #3 should suppress predicate "q".\r
+ However, the current version does not attempt to look past\r
+ the guard predicate to find context which might suppress other\r
+ predicates.\r
+\r
+ In some cases -mrhoist will lead to the reporting of ambiguities\r
+ which were not visible before:\r
+\r
+ start : (a)* "@";\r
+ a : bc | d;\r
+ bc : b | c ;\r
+\r
+ b : <<isUpper(LATEXT(1))>>? A;\r
+ c : A ;\r
+\r
+ d : A ;\r
+\r
+ In this case there is a true ambiguity in "a" between "bc" and "d"\r
+ which can both match "A". Without -mrhoist the predicate in "b"\r
+ is hoisted into "a" and there is no ambiguity reported. However,\r
+ with -mrhoist, the predicate in "b" is suppressed by "c" (as it\r
+ should be) making the ambiguity in "a" apparent.\r
+\r
+ The motivations for these changes were hoisting problems reported\r
+ by Reinier van den Born (reinier@vnet.ibm.com) and several others.\r
+\r
+#116. (Changed in 1.33MR10) C++ mode: tracein/traceout rule name is (const char *)\r
+\r
+ The prototype for C++ mode routine tracein (and traceout) has changed from\r
+ "char *" to "const char *".\r
+\r
+#115. (Changed in 1.33MR10) Using guess mode with exception handlers in C mode\r
+\r
+ The definition of the C mode macros zzmatch_wsig and zzsetmatch_wsig\r
+ neglected to consider guess mode. When control passed to the rule's\r
+ parse exception handler the routine would exit without ever closing the\r
+ guess block. This would lead to unpredictable behavior.\r
+\r
+ In 1.33MR10 the behavior of exceptions in C mode and C++ mode should be\r
+ identical.\r
+\r
+#114. (Changed in 1.33MR10) difference in [zz]resynch() between C and C++ modes\r
+\r
+ There was a slight difference in the way C and C++ mode resynchronized\r
+ following a parsing error. The C routine would sometimes skip an extra\r
+ token before attempting to resynchronize.\r
+\r
+ The C routine was changed to match the C++ routine.\r
+\r
+#113. (Changed in 1.33MR10) new context guarded pred: (g)? && <<p>>? expr\r
+\r
+ The existing context guarded predicate:\r
+\r
+ rule : (guard)? => <<p>>? expr\r
+ | next_alternative\r
+ ;\r
+\r
+ generates code which resembles:\r
+\r
+ if (lookahead(expr) && (!guard || pred)) {\r
+ expr()\r
+ } else ....\r
+\r
+ This is not suitable for some applications because it allows\r
+ expr() to be invoked when the predicate is false. This is\r
+ intentional because it is meant to mimic automatically computed\r
+ predicate context.\r
+\r
+ The new context guarded predicate uses the guard information\r
+ differently because it has a different goal. Consider:\r
+\r
+ rule : (guard)? && <<p>>? expr\r
+ | next_alternative\r
+ ;\r
+\r
+ The new style of context guarded predicate is equivalent to:\r
+\r
+ rule : <<guard==true && pred>>? expr\r
+ | next_alternative\r
+ ;\r
+\r
+ It generates code which resembles:\r
+\r
+ if (lookahead(expr) && guard && pred) {\r
+ expr();\r
+ } else ...\r
+\r
+ Both forms of guarded predicates severely restrict the form of\r
+ the context guard: it can contain no rule references, no\r
+ (...)*, no (...)+, and no {...}. It may contain token and\r
+ token class references, and alternation ("|").\r
+\r
+ Addition for 1.33MR11: in the token expression all tokens must\r
+ be at the same height of the token tree:\r
+\r
+ (A ( B | C))? && ... is ok (all height 2)\r
+ (A ( B | ))? && ... is not ok (some 1, some 2)\r
+ (A B C D | E F G H)? && ... is ok (all height 4)\r
+ (A B C D | E )? && ... is not ok (some 4, some 1)\r
+\r
+ This restriction is required in order to properly compute the lookahead\r
+ set for expressions like:\r
+\r
+ rule1 : (A B C)? && <<pred>>? rule2 ;\r
+ rule2 : (A|X) (B|Y) (C|Z);\r
+\r
+ This addition was suggested by Rienier van den Born (reinier@vnet.ibm.com)\r
+\r
+#112. (Changed in 1.33MR10) failed validation predicate in C guess mode\r
+\r
+ John Lilley (jlilley@empathy.com) suggested that failed validation\r
+ predicates abort a guess rather than reporting a failed error.\r
+ This was installed in C++ mode (Item #4). Only now was it noticed\r
+ that the fix was never installed for C mode.\r
+\r
+#111. (Changed in 1.33MR10) moved zzTRACEIN to before init action\r
+\r
+ When the antlr -gd switch is present antlr generates calls to\r
+ zzTRACEIN at the start of a rule and zzTRACEOUT at the exit\r
+ from a rule. Prior to 1.33MR10 Tthe call to zzTRACEIN was\r
+ after the init-action, which could cause confusion because the\r
+ init-actions were reported with the name of the enclosing rule,\r
+ rather than the active rule.\r
+\r
+#110. (Changed in 1.33MR10) antlr command line copied to generated file\r
+\r
+ The antlr command line is now copied to the generated file near\r
+ the start.\r
+\r
+#109. (Changed in 1.33MR10) improved trace information\r
+\r
+ The quality of the trace information provided by the "-gd"\r
+ switch has been improved significantly. Here is an example\r
+ of the output from a test program. It shows the rule name,\r
+ the first token of lookahead, the call depth, and the guess\r
+ status:\r
+\r
+ exit rule gusxx {"?"} depth 2\r
+ enter rule gusxx {"?"} depth 2\r
+ enter rule gus1 {"o"} depth 3 guessing\r
+ guess done - returning to rule gus1 {"o"} at depth 3\r
+ (guess mode continues - an enclosing guess is still active)\r
+ guess done - returning to rule gus1 {"Z"} at depth 3\r
+ (guess mode continues - an enclosing guess is still active)\r
+ exit rule gus1 {"Z"} depth 3 guessing\r
+ guess done - returning to rule gusxx {"o"} at depth 2 (guess mode ends)\r
+ enter rule gus1 {"o"} depth 3\r
+ guess done - returning to rule gus1 {"o"} at depth 3 (guess mode ends)\r
+ guess done - returning to rule gus1 {"Z"} at depth 3 (guess mode ends)\r
+ exit rule gus1 {"Z"} depth 3\r
+ line 1: syntax error at "Z" missing SC\r
+ ...\r
+\r
+ Rule trace reporting is controlled by the value of the integer\r
+ [zz]traceOptionValue: when it is positive tracing is enabled,\r
+ otherwise it is disabled. Tracing during guess mode is controlled\r
+ by the value of the integer [zz]traceGuessOptionValue. When\r
+ it is positive AND [zz]traceOptionValue is positive rule trace\r
+ is reported in guess mode.\r
+\r
+ The values of [zz]traceOptionValue and [zz]traceGuessOptionValue\r
+ can be adjusted by subroutine calls listed below.\r
+\r
+ Depending on the presence or absence of the antlr -gd switch\r
+ the variable [zz]traceOptionValueDefault is set to 0 or 1. When\r
+ the parser is initialized or [zz]traceReset() is called the\r
+ value of [zz]traceOptionValueDefault is copied to [zz]traceOptionValue.\r
+ The value of [zz]traceGuessOptionValue is always initialzed to 1,\r
+ but, as noted earlier, nothing will be reported unless\r
+ [zz]traceOptionValue is also positive.\r
+\r
+ When the parser state is saved/restored the value of the trace\r
+ variables are also saved/restored. If a restore causes a change in\r
+ reporting behavior from on to off or vice versa this will be reported.\r
+\r
+ When the -gd option is selected, the macro "#define zzTRACE_RULES"\r
+ is added to appropriate output files.\r
+\r
+ C++ mode\r
+ --------\r
+ int traceOption(int delta)\r
+ int traceGuessOption(int delta)\r
+ void traceReset()\r
+ int traceOptionValueDefault\r
+\r
+ C mode\r
+ --------\r
+ int zzTraceOption(int delta)\r
+ int zzTraceGuessOption(int delta)\r
+ void zzTraceReset()\r
+ int zzTraceOptionValueDefault\r
+\r
+ The argument "delta" is added to the traceOptionValue. To\r
+ turn on trace when inside a particular rule one:\r
+\r
+ rule : <<traceOption(+1);>>\r
+ (\r
+ rest-of-rule\r
+ )\r
+ <<traceOption(-1);>>\r
+ ; /* fail clause */ <<traceOption(-1);>>\r
+\r
+ One can use the same idea to turn *off* tracing within a\r
+ rule by using a delta of (-1).\r
+\r
+ An improvement in the rule trace was suggested by Sramji\r
+ Ramanathan (ps@kumaran.com).\r
+\r
+#108. A Note on Deallocation of Variables Allocated in Guess Mode\r
+\r
+ NOTE\r
+ ------------------------------------------------------\r
+ This mechanism only works for heap allocated variables\r
+ ------------------------------------------------------\r
+\r
+ The rewrite of the trace provides the machinery necessary\r
+ to properly free variables or undo actions following a\r
+ failed guess.\r
+\r
+ The macro zzUSER_GUESS_HOOK(guessSeq,zzrv) is expanded\r
+ as part of the zzGUESS macro. When a guess is opened\r
+ the value of zzrv is 0. When a longjmp() is executed to\r
+ undo the guess, the value of zzrv will be 1.\r
+\r
+ The macro zzUSER_GUESS_DONE_HOOK(guessSeq) is expanded\r
+ as part of the zzGUESS_DONE macro. This is executed\r
+ whether the guess succeeds or fails as part of closing\r
+ the guess.\r
+\r
+ The guessSeq is a sequence number which is assigned to each\r
+ guess and is incremented by 1 for each guess which becomes\r
+ active. It is needed by the user to associate the start of\r
+ a guess with the failure and/or completion (closing) of a\r
+ guess.\r
+\r
+ Guesses are nested. They must be closed in the reverse\r
+ of the order that they are opened.\r
+\r
+ In order to free memory used by a variable during a guess\r
+ a user must write a routine which can be called to\r
+ register the variable along with the current guess sequence\r
+ number provided by the zzUSER_GUESS_HOOK macro. If the guess\r
+ fails, all variables tagged with the corresponding guess\r
+ sequence number should be released. This is ugly, but\r
+ it would require a major rewrite of antlr 1.33 to use\r
+ some mechanism other than setjmp()/longjmp().\r
+\r
+ The order of calls for a *successful* guess would be:\r
+\r
+ zzUSER_GUESS_HOOK(guessSeq,0);\r
+ zzUSER_GUESS_DONE_HOOK(guessSeq);\r
+\r
+ The order of calls for a *failed* guess would be:\r
+\r
+ zzUSER_GUESS_HOOK(guessSeq,0);\r
+ zzUSER_GUESS_HOOK(guessSeq,1);\r
+ zzUSER_GUESS_DONE_HOOK(guessSeq);\r
+\r
+ The default definitions of these macros are empty strings.\r
+\r
+ Here is an example in C++ mode. The zzUSER_GUESS_HOOK and\r
+ zzUSER_GUESS_DONE_HOOK macros and myGuessHook() routine\r
+ can be used without change in both C and C++ versions.\r
+\r
+ ----------------------------------------------------------------------\r
+ <<\r
+\r
+ #include "AToken.h"\r
+\r
+ typedef ANTLRCommonToken ANTLRToken;\r
+\r
+ #include "DLGLexer.h"\r
+\r
+ int main() {\r
+\r
+ {\r
+ DLGFileInput in(stdin);\r
+ DLGLexer lexer(&in,2000);\r
+ ANTLRTokenBuffer pipe(&lexer,1);\r
+ ANTLRCommonToken aToken;\r
+ P parser(&pipe);\r
+\r
+ lexer.setToken(&aToken);\r
+ parser.init();\r
+ parser.start();\r
+ };\r
+\r
+ fclose(stdin);\r
+ fclose(stdout);\r
+ return 0;\r
+ }\r
+\r
+ >>\r
+\r
+ <<\r
+ char *s=NULL;\r
+\r
+ #undef zzUSER_GUESS_HOOK\r
+ #define zzUSER_GUESS_HOOK(guessSeq,zzrv) myGuessHook(guessSeq,zzrv);\r
+ #undef zzUSER_GUESS_DONE_HOOK\r
+ #define zzUSER_GUESS_DONE_HOOK(guessSeq) myGuessHook(guessSeq,2);\r
+\r
+ void myGuessHook(int guessSeq,int zzrv) {\r
+ if (zzrv == 0) {\r
+ fprintf(stderr,"User hook: starting guess #%d\n",guessSeq);\r
+ } else if (zzrv == 1) {\r
+ free (s);\r
+ s=NULL;\r
+ fprintf(stderr,"User hook: failed guess #%d\n",guessSeq);\r
+ } else if (zzrv == 2) {\r
+ free (s);\r
+ s=NULL;\r
+ fprintf(stderr,"User hook: ending guess #%d\n",guessSeq);\r
+ };\r
+ }\r
+\r
+ >>\r
+\r
+ #token A "a"\r
+ #token "[\t \ \n]" <<skip();>>\r
+\r
+ class P {\r
+\r
+ start : (top)+\r
+ ;\r
+\r
+ top : (which) ? <<fprintf(stderr,"%s is a which\n",s); free(s); s=NULL; >>\r
+ | other <<fprintf(stderr,"%s is an other\n",s); free(s); s=NULL; >>\r
+ ; <<if (s != NULL) free(s); s=NULL; >>\r
+\r
+ which : which2\r
+ ;\r
+\r
+ which2 : which3\r
+ ;\r
+ which3\r
+ : (label)? <<fprintf(stderr,"%s is a label\n",s);>>\r
+ | (global)? <<fprintf(stderr,"%s is a global\n",s);>>\r
+ | (exclamation)? <<fprintf(stderr,"%s is an exclamation\n",s);>>\r
+ ;\r
+\r
+ label : <<s=strdup(LT(1)->getText());>> A ":" ;\r
+\r
+ global : <<s=strdup(LT(1)->getText());>> A "::" ;\r
+\r
+ exclamation : <<s=strdup(LT(1)->getText());>> A "!" ;\r
+\r
+ other : <<s=strdup(LT(1)->getText());>> "other" ;\r
+\r
+ }\r
+ ----------------------------------------------------------------------\r
+\r
+ This is a silly example, but illustrates the idea. For the input\r
+ "a ::" with tracing enabled the output begins:\r
+\r
+ ----------------------------------------------------------------------\r
+ enter rule "start" depth 1\r
+ enter rule "top" depth 2\r
+ User hook: starting guess #1\r
+ enter rule "which" depth 3 guessing\r
+ enter rule "which2" depth 4 guessing\r
+ enter rule "which3" depth 5 guessing\r
+ User hook: starting guess #2\r
+ enter rule "label" depth 6 guessing\r
+ guess failed\r
+ User hook: failed guess #2\r
+ guess done - returning to rule "which3" at depth 5 (guess mode continues\r
+ - an enclosing guess is still active)\r
+ User hook: ending guess #2\r
+ User hook: starting guess #3\r
+ enter rule "global" depth 6 guessing\r
+ exit rule "global" depth 6 guessing\r
+ guess done - returning to rule "which3" at depth 5 (guess mode continues\r
+ - an enclosing guess is still active)\r
+ User hook: ending guess #3\r
+ enter rule "global" depth 6 guessing\r
+ exit rule "global" depth 6 guessing\r
+ exit rule "which3" depth 5 guessing\r
+ exit rule "which2" depth 4 guessing\r
+ exit rule "which" depth 3 guessing\r
+ guess done - returning to rule "top" at depth 2 (guess mode ends)\r
+ User hook: ending guess #1\r
+ enter rule "which" depth 3\r
+ .....\r
+ ----------------------------------------------------------------------\r
+\r
+ Remember:\r
+\r
+ (a) Only init-actions are executed during guess mode.\r
+ (b) A rule can be invoked multiple times during guess mode.\r
+ (c) If the guess succeeds the rule will be called once more\r
+ without guess mode so that normal actions will be executed.\r
+ This means that the init-action might need to distinguish\r
+ between guess mode and non-guess mode using the variable\r
+ [zz]guessing.\r
+\r
+#107. (Changed in 1.33MR10) construction of ASTs in guess mode\r
+\r
+ Prior to 1.33MR10, when using automatic AST construction in C++\r
+ mode for a rule, an AST would be constructed for elements of the\r
+ rule even while in guess mode. In MR10 this no longer occurs.\r
+\r
+#106. (Changed in 1.33MR10) guess variable confusion\r
+\r
+ In C++ mode a guess which failed always restored the parser state\r
+ using zzGUESS_DONE as part of zzGUESS_FAIL. Prior to 1.33MR10,\r
+ C mode required an explicit call to zzGUESS_DONE after the\r
+ call to zzGUESS_FAIL.\r
+\r
+ Consider:\r
+\r
+ rule : (alpha)? beta\r
+ | ...\r
+ ;\r
+\r
+ The generated code resembles:\r
+\r
+ zzGUESS\r
+ if (!zzrv && LA(1)==ID) { <==== line #1\r
+ alpha\r
+ zzGUESS_DONE\r
+ beta\r
+ } else {\r
+ if (! zzrv) zzGUESS_DONE <==== line #2a\r
+ ....\r
+\r
+ However, in some cases line #2 was rendered:\r
+\r
+ if (guessing) zzGUESS_DONE <==== line #2b\r
+\r
+ This would work for simple test cases, but would fail in\r
+ some cases where there was a guess while another guess was active.\r
+ One kind of failure would be to match up the zzGUESS_DONE at line\r
+ #2b with the "outer" guess which was still active. The outer\r
+ guess would "succeed" when only the inner guess should have\r
+ succeeded.\r
+\r
+ In 1.33MR10 the behavior of zzGUESS and zzGUESS_FAIL in C and\r
+ and C++ mode should be identical.\r
+\r
+ The same problem appears in 1.33 vanilla in some places. For\r
+ example:\r
+\r
+ start : { (sub)? } ;\r
+\r
+ or:\r
+\r
+ start : (\r
+ B\r
+ | ( sub )?\r
+ | C\r
+ )+\r
+ ;\r
+\r
+ generates incorrect code.\r
+\r
+ The general principle is:\r
+\r
+ (a) use [zz]guessing only when deciding between a call to zzFAIL\r
+ or zzGUESS_FAIL\r
+\r
+ (b) use zzrv in all other cases\r
+\r
+ This problem was discovered while testing changes to item #105.\r
+ I believe this is now fixed. My apologies.\r
+\r
+#105. (Changed in 1.33MR10) guess block as single alt of (...)+\r
+\r
+ Prior to 1.33MR10 the following constructs:\r
+\r
+ rule_plus : (\r
+ (sub)?\r
+ )+\r
+ ;\r
+\r
+ rule_star : (\r
+ (sub)?\r
+ )*\r
+ ;\r
+\r
+ generated incorrect code for the guess block (which could result\r
+ in runtime errors) because of an incorrect optimization of a\r
+ block with only a single alternative.\r
+\r
+ The fix caused some changes to the fix described in Item #49\r
+ because there are now three code generation sequences for (...)+\r
+ blocks containing a guess block:\r
+\r
+ a. single alternative which is a guess block\r
+ b. multiple alternatives in which the last is a guess block\r
+ c. all other cases\r
+\r
+ Forms like "rule_star" can have unexpected behavior when there\r
+ is a syntax error: if the subrule "sub" is not matched *exactly*\r
+ then "rule_star" will consume no tokens.\r
+\r
+ Reported by Esa Pulkkinen (esap@cs.tut.fi).\r
+\r
+#104. (Changed in 1.33MR10) -o option for dlg\r
+\r
+ There was problem with the code added by item #74 to handle the\r
+ -o option of dlg. This should fix it.\r
+\r
+#103. (Changed in 1.33MR10) ANDed semantic predicates\r
+\r
+ Rescinded.\r
+\r
+ The optimization was a mistake.\r
+ The resulting problem is described in Item #150.\r
+\r
+#102. (Changed in 1.33MR10) allow "class parser : .... {"\r
+\r
+ The syntax of the class statement ("class parser-name {")\r
+ has been extended to allow for the specification of base\r
+ classes. An arbirtrary number of tokens may now appear\r
+ between the class name and the "{". They are output\r
+ again when the class declaration is generated. For\r
+ example:\r
+\r
+ class Parser : public MyBaseClassANTLRparser {\r
+\r
+ This was suggested by a user, but I don't have a record\r
+ of who it was.\r
+\r
+#101. (Changed in 1.33MR10) antlr -info command line switch\r
+\r
+ -info\r
+\r
+ p - extra predicate information in generated file\r
+\r
+ t - information about tnode use:\r
+ at the end of each rule in generated file\r
+ summary on stderr at end of program\r
+\r
+ m - monitor progress\r
+ prints name of each rule as it is started\r
+ flushes output at start of each rule\r
+\r
+ f - first/follow set information to stdout\r
+\r
+ 0 - no operation (added in 1.33MR11)\r
+\r
+ The options may be combined and may appear in any order.\r
+ For example:\r
+\r
+ antlr -info ptm -CC -gt -mrhoist on mygrammar.g\r
+\r
+#100a. (Changed in 1.33MR10) Predicate tree simplification\r
+\r
+ When the same predicates can be referenced in more than one\r
+ alternative of a block large predicate trees can be formed.\r
+\r
+ The difference that these optimizations make is so dramatic\r
+ that I have decided to use it even when -mrhoist is not selected.\r
+\r
+ Consider the following grammar:\r
+\r
+ start : ( all )* ;\r
+\r
+ all : a\r
+ | d\r
+ | e\r
+ | f\r
+ ;\r
+\r
+ a : c A B\r
+ | c A C\r
+ ;\r
+\r
+ c : <<AAA(LATEXT(2))>>?\r
+ ;\r
+\r
+ d : <<BBB(LATEXT(2))>>? B C\r
+ ;\r
+\r
+ e : <<CCC(LATEXT(2))>>? B C\r
+ ;\r
+\r
+ f : e X Y\r
+ ;\r
+\r
+ In rule "a" there is a reference to rule "c" in both alternatives.\r
+ The length of the predicate AAA is k=2 and it can be followed in\r
+ alternative 1 only by (A B) while in alternative 2 it can be\r
+ followed only by (A C). Thus they do not have identical context.\r
+\r
+ In rule "all" the alternatives which refer to rules "e" and "f" allow\r
+ elimination of the duplicate reference to predicate CCC.\r
+\r
+ The table below summarized the kind of simplification performed by\r
+ 1.33MR10. In the table, X and Y stand for single predicates\r
+ (not trees).\r
+\r
+ (OR X (OR Y (OR Z))) => (OR X Y Z)\r
+ (AND X (AND Y (AND Z))) => (AND X Y Z)\r
+\r
+ (OR X (... (OR X Y) ... )) => (OR X (... Y ... ))\r
+ (AND X (... (AND X Y) ... )) => (AND X (... Y ... ))\r
+ (OR X (... (AND X Y) ... )) => (OR X (... ... ))\r
+ (AND X (... (OR X Y) ... )) => (AND X (... ... ))\r
+\r
+ (AND X) => X\r
+ (OR X) => X\r
+\r
+ In a test with a complex grammar for a real application, a predicate\r
+ tree with six OR nodes and 12 leaves was reduced to "(OR X Y Z)".\r
+\r
+ In 1.33MR10 there is a greater effort to release memory used\r
+ by predicates once they are no longer in use.\r
+\r
+#100b. (Changed in 1.33MR10) Suppression of extra predicate tests\r
+\r
+ The following optimizations require that -mrhoist be selected.\r
+\r
+ It is relatively easy to optimize the code generated for predicate\r
+ gates when they are of the form:\r
+\r
+ (AND X Y Z ...)\r
+ or (OR X Y Z ...)\r
+\r
+ where X, Y, Z, and "..." represent individual predicates (leaves) not\r
+ predicate trees.\r
+\r
+ If the predicate is an AND the contexts of the X, Y, Z, etc. are\r
+ ANDed together to create a single Tree context for the group and\r
+ context tests for the individual predicates are suppressed:\r
+\r
+ --------------------------------------------------\r
+ Note: This was incorrect. The contexts should be\r
+ ORed together. This has been fixed. A more \r
+ complete description is available in item #152.\r
+ ---------------------------------------------------\r
+\r
+ Optimization 1: (AND X Y Z ...)\r
+\r
+ Suppose the context for Xtest is LA(1)==LP and the context for\r
+ Ytest is LA(1)==LP && LA(2)==ID.\r
+\r
+ Without the optimization the code would resemble:\r
+\r
+ if (lookaheadContext &&\r
+ !(LA(1)==LP && LA(1)==LP && LA(2)==ID) ||\r
+ ( (! LA(1)==LP || Xtest) &&\r
+ (! (LA(1)==LP || LA(2)==ID) || Xtest)\r
+ )) {...\r
+\r
+ With the -mrhoist optimization the code would resemble:\r
+\r
+ if (lookaheadContext &&\r
+ ! (LA(1)==LP && LA(2)==ID) || (Xtest && Ytest) {...\r
+\r
+ Optimization 2: (OR X Y Z ...) with identical contexts\r
+\r
+ Suppose the context for Xtest is LA(1)==ID and for Ytest\r
+ the context is also LA(1)==ID.\r
+\r
+ Without the optimization the code would resemble:\r
+\r
+ if (lookaheadContext &&\r
+ ! (LA(1)==ID || LA(1)==ID) ||\r
+ (LA(1)==ID && Xtest) ||\r
+ (LA(1)==ID && Ytest) {...\r
+\r
+ With the -mrhoist optimization the code would resemble:\r
+\r
+ if (lookaheadContext &&\r
+ (! LA(1)==ID) || (Xtest || Ytest) {...\r
+\r
+ Optimization 3: (OR X Y Z ...) with distinct contexts\r
+\r
+ Suppose the context for Xtest is LA(1)==ID and for Ytest\r
+ the context is LA(1)==LP.\r
+\r
+ Without the optimization the code would resemble:\r
+\r
+ if (lookaheadContext &&\r
+ ! (LA(1)==ID || LA(1)==LP) ||\r
+ (LA(1)==ID && Xtest) ||\r
+ (LA(1)==LP && Ytest) {...\r
+\r
+ With the -mrhoist optimization the code would resemble:\r
+\r
+ if (lookaheadContext &&\r
+ (zzpf=0,\r
+ (LA(1)==ID && (zzpf=1) && Xtest) ||\r
+ (LA(1)==LP && (zzpf=1) && Ytest) ||\r
+ !zzpf) {\r
+\r
+ These may appear to be of similar complexity at first,\r
+ but the non-optimized version contains two tests of each\r
+ context while the optimized version contains only one\r
+ such test, as well as eliminating some of the inverted\r
+ logic (" !(...) || ").\r
+\r
+ Optimization 4: Computation of predicate gate trees\r
+\r
+ When generating code for the gates of predicate expressions\r
+ antlr 1.33 vanilla uses a recursive procedure to generate\r
+ "&&" and "||" expressions for testing the lookahead. As each\r
+ layer of the predicate tree is exposed a new set of "&&" and\r
+ "||" expressions on the lookahead are generated. In many\r
+ cases the lookahead being tested has already been tested.\r
+\r
+ With -mrhoist a lookahead tree is computed for the entire\r
+ lookahead expression. This means that predicates with identical\r
+ context or context which is a subset of another predicate's\r
+ context disappear.\r
+\r
+ This is especially important for predicates formed by rules\r
+ like the following:\r
+\r
+ uppperCaseVowel : <<isUpperCase(LATEXT(1))>>? vowel;\r
+ vowel: : <<isVowel(LATEXT(1))>>? LETTERS;\r
+\r
+ These predicates are combined using AND since both must be\r
+ satisfied for rule upperCaseVowel. They have identical\r
+ context which makes this optimization very effective.\r
+\r
+ The affect of Items #100a and #100b together can be dramatic. In\r
+ a very large (but real world) grammar one particular predicate\r
+ expression was reduced from an (unreadable) 50 predicate leaves,\r
+ 195 LA(1) terms, and 5500 characters to an (easily comprehensible)\r
+ 3 predicate leaves (all different) and a *single* LA(1) term.\r
+\r
+#99. (Changed in 1.33MR10) Code generation for expression trees\r
+\r
+ Expression trees are used for k>1 grammars and predicates with\r
+ lookahead depth >1. This optimization must be enabled using\r
+ "-mrhoist on". (Clarification added for 1.33MR11).\r
+\r
+ In the processing of expression trees, antlr can generate long chains\r
+ of token comparisons. Prior to 1.33MR10 there were many redundant\r
+ parenthesis which caused problems for compilers which could handle\r
+ expressions of only limited complexity. For example, to test an\r
+ expression tree (root R A B C D), antlr would generate something\r
+ resembling:\r
+\r
+ (LA(1)==R && (LA(2)==A || (LA(2)==B || (LA(2)==C || LA(2)==D)))))\r
+\r
+ If there were twenty tokens to test then there would be twenty\r
+ parenthesis at the end of the expression.\r
+\r
+ In 1.33MR10 the generated code for tree expressions resembles:\r
+\r
+ (LA(1)==R && (LA(2)==A || LA(2)==B || LA(2)==C || LA(2)==D))\r
+\r
+ For "complex" expressions the output is indented to reflect the LA\r
+ number being tested:\r
+\r
+ (LA(1)==R\r
+ && (LA(2)==A || LA(2)==B || LA(2)==C || LA(2)==D\r
+ || LA(2)==E || LA(2)==F)\r
+ || LA(1)==S\r
+ && (LA(2)==G || LA(2)==H))\r
+\r
+\r
+ Suggested by S. Bochnak (S.Bochnak@@microTool.com.pl),\r
+\r
+#98. (Changed in 1.33MR10) Option "-info p"\r
+\r
+ When the user selects option "-info p" the program will generate\r
+ detailed information about predicates. If the user selects\r
+ "-mrhoist on" additional detail will be provided explaining\r
+ the promotion and suppression of predicates. The output is part\r
+ of the generated file and sandwiched between #if 0/#endif statements.\r
+\r
+ Consider the following k=1 grammar:\r
+\r
+ start : ( all ) * ;\r
+\r
+ all : ( a\r
+ | b\r
+ )\r
+ ;\r
+\r
+ a : c B\r
+ ;\r
+\r
+ c : <<LATEXT(1)>>?\r
+ | B\r
+ ;\r
+\r
+ b : <<LATEXT(1)>>? X\r
+ ;\r
+\r
+ Below is an excerpt of the output for rule "start" for the three\r
+ predicate options (off, on, and maintenance release style hoisting).\r
+\r
+ For those who do not wish to use the "-mrhoist on" option for code\r
+ generation the option can be used in a "diagnostic" mode to provide\r
+ valuable information:\r
+\r
+ a. where one should insert null actions to inhibit hoisting\r
+ b. a chain of rule references which shows where predicates are\r
+ being hoisted\r
+\r
+ ======================================================================\r
+ Example of "-info p" with "-mrhoist on"\r
+ ======================================================================\r
+ #if 0\r
+\r
+ Hoisting of predicate suppressed by alternative without predicate.\r
+ The alt without the predicate includes all cases where the\r
+ predicate is false.\r
+\r
+ WITH predicate: line 11 v36.g\r
+ WITHOUT predicate: line 12 v36.g\r
+\r
+ The context set for the predicate:\r
+\r
+ B\r
+\r
+ The lookahead set for alt WITHOUT the semantic predicate:\r
+\r
+ B\r
+\r
+ The predicate:\r
+\r
+ pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g\r
+\r
+ set context:\r
+ B\r
+ tree context: null\r
+\r
+ Chain of referenced rules:\r
+\r
+ #0 in rule start (line 1 v36.g) to rule all\r
+ #1 in rule all (line 3 v36.g) to rule a\r
+ #2 in rule a (line 8 v36.g) to rule c\r
+ #3 in rule c (line 11 v36.g)\r
+\r
+ #endif\r
+ &&\r
+ #if 0\r
+\r
+ pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g\r
+\r
+ set context:\r
+ X\r
+ tree context: null\r
+\r
+ #endif\r
+ ======================================================================\r
+ Example of "-info p" with the default -prc setting ( "-prc off")\r
+ ======================================================================\r
+ #if 0\r
+\r
+ OR\r
+ pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g\r
+\r
+ set context:\r
+ nil\r
+ tree context: null\r
+\r
+ pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g\r
+\r
+ set context:\r
+ nil\r
+ tree context: null\r
+\r
+ #endif\r
+ ======================================================================\r
+ Example of "-info p" with "-prc on" and "-mrhoist off"\r
+ ======================================================================\r
+ #if 0\r
+\r
+ OR\r
+ pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g\r
+\r
+ set context:\r
+ B\r
+ tree context: null\r
+\r
+ pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g\r
+\r
+ set context:\r
+ X\r
+ tree context: null\r
+\r
+ #endif\r
+ ======================================================================\r
+\r
+#97. (Fixed in 1.33MR10) "Predicate applied for more than one ... "\r
+\r
+ In 1.33 vanilla, the grammar listed below produced this message for\r
+ the first alternative (only) of rule "b":\r
+\r
+ warning: predicate applied for >1 lookahead 1-sequences\r
+ [you may only want one lookahead 1-sequence to apply.\r
+ Try using a context guard '(...)? =>'\r
+\r
+ In 1.33MR10 the message is issued for both alternatives.\r
+\r
+ top : (a)*;\r
+ a : b | c ;\r
+\r
+ b : <<PPP(LATEXT(1))>>? ( AAA | BBB )\r
+ | <<QQQ(LATEXT(1))>>? ( XXX | YYY )\r
+ ;\r
+\r
+ c : AAA | XXX;\r
+\r
+#96. (Fixed in 1.33MR10) Guard predicates ignored when -prc off\r
+\r
+ Prior to 1.33MR10, guard predicate code was not generated unless\r
+ "-prc on" was selected.\r
+\r
+ This was incorrect, since "-prc off" (the default) is supposed to\r
+ disable only AUTOMATIC computation of predicate context, not the\r
+ programmer specified context supplied by guard predicates.\r
+\r
+#95. (Fixed in 1.33MR10) Predicate guard context length was k, not max(k,ck)\r
+\r
+ Prior to 1.33MR10, predicate guards were computed to k tokens rather\r
+ than max(k,ck). Consider the following grammar:\r
+\r
+ a : ( A B C)? => <<AAA(LATEXT(1))>>? (A|X) (B|Y) (C|Z) ;\r
+\r
+ The code generated by 1.33 vanilla with "-k 1 -ck 3 -prc on"\r
+ for the predicate in "a" resembles:\r
+\r
+ if ( (! LA(1)==A) || AAA(LATEXT(1))) {...\r
+\r
+ With 1.33MR10 and the same options the code resembles:\r
+\r
+ if ( (! (LA(1)==A && LA(2)==B && LA(3)==C) || AAA(LATEXT(1))) {...\r
+\r
+#94. (Fixed in 1.33MR10) Predicates followed by rule references\r
+\r
+ Prior to 1.33MR10, a semantic predicate which referenced a token\r
+ which was off the end of the rule caused an incomplete context\r
+ to be computed (with "-prc on") for the predicate under some circum-\r
+ stances. In some cases this manifested itself as illegal C code\r
+ (e.g. "LA(2)==[Ep](1)" in the k=2 examples below:\r
+\r
+ all : ( a ) *;\r
+\r
+ a : <<AAA(LATEXT(2))>>? ID X\r
+ | <<BBB(LATEXT(2))>>? Y\r
+ | Z\r
+ ;\r
+\r
+ This might also occur when the semantic predicate was followed\r
+ by a rule reference which was shorter than the length of the\r
+ semantic predicate:\r
+\r
+ all : ( a ) *;\r
+\r
+ a : <<AAA(LATEXT(2))>>? ID X\r
+ | <<BBB(LATEXT(2))>>? y\r
+ | Z\r
+ ;\r
+\r
+ y : Y ;\r
+\r
+ Depending on circumstance, the resulting context might be too\r
+ generous because it was too short, or too restrictive because\r
+ of missing alternatives.\r
+\r
+#93. (Changed in 1.33MR10) Definition of Purify macro\r
+\r
+ Ofer Ben-Ami (gremlin@cs.huji.ac.il) has supplied a definition\r
+ for the Purify macro:\r
+\r
+ #define PURIFY(r, s) memset((char *) &(r), '\0', (s));\r
+\r
+ Note: This may not be the right thing to do for C++ objects that\r
+ have constructors. Reported by Bonny Rais (bonny@werple.net.au).\r
+\r
+ For those cases one should #define PURIFY to an empty macro in the\r
+ #header or #first actions.\r
+\r
+#92. (Fixed in 1.33MR10) Guarded predicates and hoisting\r
+\r
+ When a guarded predicate participates in hoisting it is linked into\r
+ a predicate expression tree. Prior to 1.33MR10 this link was never\r
+ cleared and the next time the guard was used to construct a new\r
+ tree the link could contain a spurious reference to another element\r
+ which had previosly been joined to it in the semantic predicate tree.\r
+\r
+ For example:\r
+\r
+ start : ( all ) *;\r
+ all : ( a | b ) ;\r
+\r
+ start2 : ( all2 ) *;\r
+ all2 : ( a ) ;\r
+\r
+ a : (A)? => <<AAA(LATEXT(1))>>? A ;\r
+ b : (B)? => <<BBB(LATEXT(1))>>? B ;\r
+\r
+ Prior to 1.33MR10 the code for "start2" would include a spurious\r
+ reference to the BBB predicate which was left from constructing\r
+ the predicate tree for rule "start" (i.e. or(AAA,BBB) ).\r
+\r
+ In 1.33MR10 this problem is avoided by cloning the original guard\r
+ each time it is linked into a predicate tree.\r
+\r
+#91. (Changed in 1.33MR10) Extensive changes to semantic pred hoisting\r
+\r
+ ============================================\r
+ This has been rendered obsolete by Item #117\r
+ ============================================\r
+\r
+#90. (Fixed in 1.33MR10) Semantic pred with LT(i) and i>max(k,ck)\r
+\r
+ There is a bug in antlr 1.33 vanilla and all maintenance releases\r
+ prior to 1.33MR10 which allows semantic predicates to reference\r
+ an LT(i) or LATEXT(i) where i is larger than max(k,ck). When\r
+ this occurs antlr will attempt to mark the ith element of an array\r
+ in which there are only max(k,ck) elements. The result cannot\r
+ be predicted.\r
+\r
+ Using LT(i) or LATEXT(i) for i>max(k,ck) is reported as an error\r
+ in 1.33MR10.\r
+\r
+#89. Rescinded\r
+\r
+#88. (Fixed in 1.33MR10) Tokens used in semantic predicates in guess mode\r
+\r
+ Consider the behavior of a semantic predicate during guess mode:\r
+\r
+ rule : a:A (\r
+ <<test($a)>>? b:B\r
+ | c:C\r
+ );\r
+\r
+ Prior to MR10 the assignment of the token or attribute to\r
+ $a did not occur during guess mode, which would cause the\r
+ semantic predicate to misbehave because $a would be null.\r
+\r
+ In 1.33MR10 a semantic predicate with a reference to an\r
+ element label (such as $a) forces the assignment to take\r
+ place even in guess mode.\r
+\r
+ In order to work, this fix REQUIRES use of the $label format\r
+ for token pointers and attributes referenced in semantic\r
+ predicates.\r
+\r
+ The fix does not apply to semantic predicates using the\r
+ numeric form to refer to attributes (e.g. <<test($1)>>?).\r
+ The user will receive a warning for this case.\r
+\r
+ Reported by Rob Trout (trout@mcs.cs.kent.edu).\r
+\r
+#87. (Fixed in 1.33MR10) Malformed guard predicates\r
+\r
+ Context guard predicates may contain only references to\r
+ tokens. They may not contain references to (...)+ and\r
+ (...)* blocks. This is now checked. This replaces the\r
+ fatal error message in item #78 with an appropriate\r
+ (non-fatal) error messge.\r
+\r
+ In theory, context guards should be allowed to reference\r
+ rules. However, I have not had time to fix this.\r
+ Evaluation of the guard takes place before all rules have\r
+ been read, making it difficult to resolve a forward reference\r
+ to rule "zzz" - it hasn't been read yet ! To postpone evaluation\r
+ of the guard until all rules have been read is too much\r
+ for the moment.\r
+\r
+#86. (Fixed in 1.33MR10) Unequal set size in set_sub\r
+\r
+ Routine set_sub() in pccts/support/set/set.h did not work\r
+ correctly when the sets were of unequal sizes. Rewrote\r
+ set_equ to make it simpler and remove unnecessary and\r
+ expensive calls to set_deg(). This routine was not used\r
+ in 1.33 vanila.\r
+\r
+#85. (Changed in 1.33MR10) Allow redefinition of MaxNumFiles\r
+\r
+ Raised the maximum number of input files to 99 from 20.\r
+ Put a #ifndef/#endif around the "#define MaxNumFiles 99".\r
+\r
+#84. (Fixed in 1.33MR10) Initialize zzBadTok in macro zzRULE\r
+\r
+ Initialize zzBadTok to NULL in zzRULE macro of AParser.h.\r
+ in order to get rid of warning messages.\r
+\r
+#83. (Fixed in 1.33MR10) False warnings with -w2 for #tokclass\r
+\r
+ When -w2 is selected antlr gives inappropriate warnings about\r
+ #tokclass names not having any associated regular expressions.\r
+ Since a #tokclass is not a "real" token it will never have an\r
+ associated regular expression and there should be no warning.\r
+\r
+ Reported by Derek Pappas (derek.pappas@eng.sun.com)\r
+\r
+#82. (Fixed in 1.33MR10) Computation of follow sets with multiple cycles\r
+\r
+ Reinier van den Born (reinier@vnet.ibm.com) reported a problem\r
+ in the computation of follow sets by antlr. The problem (bug)\r
+ exists in 1.33 vanilla and all maintenance releases prior to 1.33MR10.\r
+\r
+ The problem involves the computation of follow sets when there are\r
+ cycles - rules which have mutual references. I believe the problem\r
+ is restricted to cases where there is more than one cycle AND\r
+ elements of those cycles have rules in common. Even when this\r
+ occurs it may not affect the code generated - but it might. It\r
+ might also lead to undetected ambiguities.\r
+\r
+ There were no changes in antlr or dlg output from the revised version.\r
+\r
+ The following fragment demonstates the problem by giving different\r
+ follow sets (option -pa) for var_access when built with k=1 and ck=2 on\r
+ 1.33 vanilla and 1.33MR10:\r
+\r
+ echo_statement : ECHO ( echo_expr )*\r
+ ;\r
+\r
+ echo_expr : ( command )?\r
+ | expression\r
+ ;\r
+\r
+ command : IDENTIFIER\r
+ { concat }\r
+ ;\r
+\r
+ expression : operand ( OPERATOR operand )*\r
+ ;\r
+\r
+ operand : value\r
+ | START command END\r
+ ;\r
+\r
+ value : concat\r
+ | TYPE operand\r
+ ;\r
+\r
+ concat : var_access { CONCAT value }\r
+ ;\r
+\r
+ var_access : IDENTIFIER { INDEX }\r
+\r
+ ;\r
+#81. (Changed in 1.33MR10) C mode use of attributes and ASTs\r
+\r
+ Reported by Isaac Clark (irclark@mindspring.com).\r
+\r
+ C mode code ignores attributes returned by rules which are\r
+ referenced using element labels when ASTs are enabled (-gt option).\r
+\r
+ 1. start : r:rule t:Token <<$start=$r;>>\r
+\r
+ The $r refrence will not work when combined with\r
+ the -gt option.\r
+\r
+ 2. start : t:Token <<$start=$t;>>\r
+\r
+ The $t reference works in all cases.\r
+\r
+ 3. start : rule <<$0=$1;>>\r
+\r
+ Numeric labels work in all cases.\r
+\r
+ With MR10 the user will receive an error message for case 1 when\r
+ the -gt option is used.\r
+\r
+#80. (Fixed in 1.33MR10) (...)? as last alternative of block\r
+\r
+ A construct like the following:\r
+\r
+ rule : a\r
+ | (b)?\r
+ ;\r
+\r
+ does not make sense because there is no alternative when\r
+ the guess block fails. This is now reported as a warning\r
+ to the user.\r
+\r
+ Previously, there was a code generation error for this case:\r
+ the guess block was not "closed" when the guess failed.\r
+ This could cause an infinite loop or other problems. This\r
+ is now fixed.\r
+\r
+ Example problem:\r
+\r
+ #header<<\r
+ #include <stdio.h>\r
+ #include "charptr.h"\r
+ >>\r
+\r
+ <<\r
+ #include "charptr.c"\r
+ main ()\r
+ {\r
+ ANTLR(start(),stdin);\r
+ }\r
+ >>\r
+\r
+ #token "[\ \t]+" << zzskip(); >>\r
+ #token "[\n]" << zzline++; zzskip(); >>\r
+\r
+ #token Word "[a-z]+"\r
+ #token Number "[0-9]+"\r
+\r
+\r
+ start : (test1)?\r
+ | (test2)?\r
+ ;\r
+ test1 : (Word Word Word Word)?\r
+ | (Word Word Word Number)?\r
+ ;\r
+ test2 : (Word Word Number Word)?\r
+ | (Word Word Number Number)?\r
+ ;\r
+\r
+ Test data which caused infinite loop:\r
+\r
+ a 1 a a\r
+\r
+#79. (Changed in 1.33MR10) Use of -fh with multiple parsers\r
+\r
+ Previously, antlr always used the pre-processor symbol\r
+ STDPCCTS_H as a gate for the file stdpccts.h. This\r
+ caused problems when there were multiple parsers defined\r
+ because they used the same gate symbol.\r
+\r
+ In 1.33MR10, the -fh filename is used to generate the\r
+ gate file for stdpccts.h. For instance:\r
+\r
+ antlr -fh std_parser1.h\r
+\r
+ generates the pre-processor symbol "STDPCCTS_std_parser1_H".\r
+\r
+ Reported by Ramanathan Santhanam (ps@kumaran.com).\r
+\r
+#78. (Changed in 1.33MR9) Guard predicates that refer to rules\r
+\r
+ ------------------------\r
+ Please refer to Item #87\r
+ ------------------------\r
+\r
+ Guard predicates are processed during an early phase\r
+ of antlr (during parsing) before all data structures\r
+ are completed.\r
+\r
+ There is an apparent bug in earlier versions of 1.33\r
+ which caused guard predicates which contained references\r
+ to rules (rather than tokens) to reference a structure\r
+ which hadn't yet been initialized.\r
+\r
+ In some cases (perhaps all cases) references to rules\r
+ in guard predicates resulted in the use of "garbage".\r
+\r
+#79. (Changed in 1.33MR9) Jeff Vincent (JVincent@novell.com)\r
+\r
+ Previously, the maximum length file name was set\r
+ arbitrarily to 300 characters in antlr, dlg, and sorcerer.\r
+\r
+ The config.h file now attempts to define the maximum length\r
+ filename using _MAX_PATH from stdlib.h before falling back\r
+ to using the value 300.\r
+\r
+#78. (Changed in 1.33MR9) Jeff Vincent (JVincent@novell.com)\r
+\r
+ Put #ifndef/#endif around definition of ZZLEXBUFSIZE in\r
+ antlr.\r
+\r
+#77. (Changed in 1.33MR9) Arithmetic overflow for very large grammars\r
+\r
+ In routine HandleAmbiguities() antlr attempts to compute the\r
+ number of possible elements in a set that is order of\r
+ number-of-tokens raised to the number-of-lookahead-tokens power.\r
+ For large grammars or large lookahead (e.g. -ck 7) this can\r
+ cause arithmetic overflow.\r
+\r
+ With 1.33MR9, arithmetic overflow in this computation is reported\r
+ the first time it happens. The program continues to run and\r
+ the program branches based on the assumption that the computed\r
+ value is larger than any number computed by counting actual cases\r
+ because 2**31 is larger than the number of bits in most computers.\r
+\r
+ Before 1.33MR9 overflow was not reported. The behavior following\r
+ overflow is not predictable by anyone but the original author.\r
+\r
+ NOTE\r
+\r
+ In 1.33MR10 the warning message is suppressed.\r
+ The code which detects the overflow allows the\r
+ computation to continue without an error. The\r
+ error message itself made made users worry.\r
+\r
+#76. (Changed in 1.33MR9) Jeff Vincent (JVincent@novell.com)\r
+\r
+ Jeff Vincent has convinced me to make ANTLRCommonToken and\r
+ ANTLRCommonNoRefCountToken use variable length strings\r
+ allocated from the heap rather than fixed length strings.\r
+ By suitable definition of setText(), the copy constructor,\r
+ and operator =() it is possible to maintain "copy" semantics.\r
+ By "copy" semantics I mean that when a token is copied from\r
+ an existing token it receives its own, distinct, copy of the\r
+ text allocated from the heap rather than simply a pointer\r
+ to the original token's text.\r
+\r
+ ============================================================\r
+ W * A * R * N * I * N * G\r
+ ============================================================\r
+\r
+ It is possible that this may cause problems for some users.\r
+ For those users I have included the old version of AToken.h as\r
+ pccts/h/AToken_traditional.h.\r
+\r
+#75. (Changed in 1.33MR9) Bruce Guenter (bruceg@qcc.sk.ca)\r
+\r
+ Make DLGStringInput const correct. Since this is infrequently\r
+ subclassed, it should affect few users, I hope.\r
+\r
+#74. (Changed in 1.33MR9) -o (output directory) option\r
+\r
+ Antlr does not properly handle the -o output directory option\r
+ when the filename of the grammar contains a directory part. For\r
+ example:\r
+\r
+ antlr -o outdir pccts_src/myfile.g\r
+\r
+ causes antlr create a file called "outdir/pccts_src/myfile.cpp.\r
+ It SHOULD create outdir/myfile.cpp\r
+\r
+ The suggested code fix has been installed in antlr, dlg, and\r
+ Sorcerer.\r
+\r
+#73. (Changed in 1.33MR9) Hoisting of semantic predicates and -mrhoist\r
+\r
+ ============================================\r
+ This has been rendered obsolete by Item #117\r
+ ============================================\r
+\r
+#72. (Changed in 1.33MR9) virtual saveState()/restoreState()/guess_XXX\r
+\r
+ The following methods in ANTLRParser were made virtual at\r
+ the request of S. Bochnak (S.Bochnak@microTool.com.pl):\r
+\r
+ saveState() and restoreState()\r
+ guess(), guess_fail(), and guess_done()\r
+\r
+#71. (Changed in 1.33MR9) Access to omitted command line argument\r
+\r
+ If a switch requiring arguments is the last thing on the\r
+ command line, and the argument is omitted, antlr would core.\r
+\r
+ antlr test.g -prc\r
+\r
+ instead of\r
+\r
+ antlr test.g -prc off\r
+\r
+#70. (Changed in 1.33MR9) Addition of MSVC .dsp and .mak build files\r
+\r
+ The following MSVC .dsp and .mak files for pccts and sorcerer\r
+ were contributed by Stanislaw Bochnak (S.Bochnak@microTool.com.pl)\r
+ and Jeff Vincent (JVincent@novell.com)\r
+\r
+ PCCTS Distribution Kit\r
+ ----------------------\r
+ pccts/PCCTSMSVC50.dsw\r
+\r
+ pccts/antlr/AntlrMSVC50.dsp\r
+ pccts/antlr/AntlrMSVC50.mak\r
+\r
+ pccts/dlg/DlgMSVC50.dsp\r
+ pccts/dlg/DlgMSVC50.mak\r
+\r
+ pccts/support/msvc.dsp\r
+\r
+ Sorcerer Distribution Kit\r
+ -------------------------\r
+ pccts/sorcerer/SorcererMSVC50.dsp\r
+ pccts/sorcerer/SorcererMSVC50.mak\r
+\r
+ pccts/sorcerer/lib/msvc.dsp\r
+\r
+#69. (Changed in 1.33MR9) Change "unsigned int" to plain "int"\r
+\r
+ Declaration of max_token_num in misc.c as "unsigned int"\r
+ caused comparison between signed and unsigned ints giving\r
+ warning message without any special benefit.\r
+\r
+#68. (Changed in 1.33MR9) Add void return for dlg internal_error()\r
+\r
+ Get rid of "no return value" message in internal_error()\r
+ in file dlg/support.c and dlg/dlg.h.\r
+\r
+#67. (Changed in Sor) sor.g: lisp() has no return value\r
+\r
+ Added a "void" for the return type.\r
+\r
+#66. (Added to Sor) sor.g: ZZLEXBUFSIZE enclosed in #ifndef/#endif\r
+\r
+ A user needed to be able to change the ZZLEXBUFSIZE for\r
+ sor. Put the definition of ZZLEXBUFSIZE inside #ifndef/#endif\r
+\r
+#65. (Changed in 1.33MR9) PCCTSAST::deepCopy() and ast_dup() bug\r
+\r
+ Jeff Vincent (JVincent@novell.com) found that deepCopy()\r
+ made new copies of only the direct descendents. No new\r
+ copies were made of sibling nodes, Sibling pointers are\r
+ set to zero by shallowCopy().\r
+\r
+ PCCTS_AST::deepCopy() has been changed to make a\r
+ deep copy in the traditional sense.\r
+\r
+ The deepCopy() routine depends on the behavior of\r
+ shallowCopy(). In all sor examples I've found,\r
+ shallowCopy() zeroes the right and down pointers.\r
+\r
+ Original Tree Original deepCopy() Revised deepCopy\r
+ ------------- ------------------- ----------------\r
+ a->b->c A A\r
+ | | |\r
+ d->e->f D D->E->F\r
+ | | |\r
+ g->h->i G G->H->I\r
+ | |\r
+ j->k J->K\r
+\r
+ While comparing deepCopy() for C++ mode with ast_dup for\r
+ C mode I found a problem with ast_dup().\r
+\r
+ Routine ast_dup() has been changed to make a deep copy\r
+ in the traditional sense.\r
+\r
+ Original Tree Original ast_dup() Revised ast_dup()\r
+ ------------- ------------------- ----------------\r
+ a->b->c A->B->C A\r
+ | | |\r
+ d->e->f D->E->F D->E->F\r
+ | | |\r
+ g->h->i G->H->I G->H->I\r
+ | | |\r
+ j->k J->K J->K\r
+\r
+\r
+ I believe this affects transform mode sorcerer programs only.\r
+\r
+#64. (Changed in 1.33MR9) anltr/hash.h prototype for killHashTable()\r
+\r
+#63. (Changed in 1.33MR8) h/charptr.h does not zero pointer after free\r
+\r
+ The charptr.h routine now zeroes the pointer after free().\r
+\r
+ Reported by Jens Tingleff (jensting@imaginet.fr)\r
+\r
+#62. (Changed in 1.33MR8) ANTLRParser::resynch had static variable\r
+\r
+ The static variable "consumed" in ANTLRParser::resynch was\r
+ changed into an instance variable of the class with the\r
+ name "resynchConsumed".\r
+\r
+ Reported by S.Bochnak@microTool.com.pl\r
+\r
+#61. (Changed in 1.33MR8) Using rule>[i,j] when rule has no return values\r
+\r
+ Previously, the following code would cause antlr to core when\r
+ it tried to generate code for rule1 because rule2 had no return\r
+ values ("upward inheritance"):\r
+\r
+ rule1 : <<int i; int j>>\r
+ rule2 > [i,j]\r
+ ;\r
+\r
+ rule2 : Anything ;\r
+\r
+ Reported by S.Bochnak@microTool.com.pl\r
+\r
+ Verified correct operation of antlr MR8 when missing or extra\r
+ inheritance arguments for all combinations. When there are\r
+ missing or extra arguments code will still be generated even\r
+ though this might cause the invocation of a subroutine with\r
+ the wrong number of arguments.\r
+\r
+#60. (Changed in 1.33MR7) Major changes to exception handling\r
+\r
+ There were significant problems in the handling of exceptions\r
+ in 1.33 vanilla. The general problem is that it can only\r
+ process one level of exception handler. For example, a named\r
+ exception handler, an exception handler for an alternative, or\r
+ an exception for a subrule always went to the rule's exception\r
+ handler if there was no "catch" which matched the exception.\r
+\r
+ In 1.33MR7 the exception handlers properly "nest". If an\r
+ exception handler does not have a matching "catch" then the\r
+ nextmost outer exception handler is checked for an appropriate\r
+ "catch" clause, and so on until an exception handler with an\r
+ appropriate "catch" is found.\r
+\r
+ There are still undesirable features in the way exception\r
+ handlers are implemented, but I do not have time to fix them\r
+ at the moment:\r
+\r
+ The exception handlers for alternatives are outside the\r
+ block containing the alternative. This makes it impossible\r
+ to access variables declared in a block or to resume the\r
+ parse by "falling through". The parse can still be easily\r
+ resumed in other ways, but not in the most natural fashion.\r
+\r
+ This results in an inconsistentcy between named exception\r
+ handlers and exception handlers for alternatives. When\r
+ an exception handler for an alternative "falls through"\r
+ it goes to the nextmost outer handler - not the "normal\r
+ action".\r
+\r
+ A major difference between 1.33MR7 and 1.33 vanilla is\r
+ the default action after an exception is caught:\r
+\r
+ 1.33 Vanilla\r
+ ------------\r
+ In 1.33 vanilla the signal value is set to zero ("NoSignal")\r
+ and the code drops through to the code following the exception.\r
+ For named exception handlers this is the "normal action".\r
+ For alternative exception handlers this is the rule's handler.\r
+\r
+ 1.33MR7\r
+ -------\r
+ In 1.33MR7 the signal value is NOT automatically set to zero.\r
+\r
+ There are two cases:\r
+\r
+ For named exception handlers: if the signal value has been\r
+ set to zero the code drops through to the "normal action".\r
+\r
+ For all other cases the code branches to the nextmost outer\r
+ exception handler until it reaches the handler for the rule.\r
+\r
+ The following macros have been defined for convenience:\r
+\r
+ C/C++ Mode Name\r
+ --------------------\r
+ (zz)suppressSignal\r
+ set signal & return signal arg to 0 ("NoSignal")\r
+ (zz)setSignal(intValue)\r
+ set signal & return signal arg to some value\r
+ (zz)exportSignal\r
+ copy the signal value to the return signal arg\r
+\r
+ I'm not sure why PCCTS make a distinction between the local\r
+ signal value and the return signal argument, but I'm loathe\r
+ to change the code. The burden of copying the local signal\r
+ value to the return signal argument can be given to the\r
+ default signal handler, I suppose.\r
+\r
+#59. (Changed in 1.33MR7) Prototypes for some functions\r
+\r
+ Added prototypes for the following functions to antlr.h\r
+\r
+ zzconsumeUntil()\r
+ zzconsumeUntilToken()\r
+\r
+#58. (Changed in 1.33MR7) Added defintion of zzbufsize to dlgauto.h\r
+\r
+#57. (Changed in 1.33MR7) Format of #line directive\r
+\r
+ Previously, the -gl directive for line 1234 would\r
+ resemble: "# 1234 filename.g". This caused problems\r
+ for some compilers/pre-processors. In MR7 it generates\r
+ "#line 1234 filename.g".\r
+\r
+#56. (Added in 1.33MR7) Jan Mikkelsen <janm@zeta.org.au>\r
+\r
+ Move PURIFY macro invocaton to after rule's init action.\r
+\r
+#55. (Fixed in 1.33MR7) Unitialized variables in ANTLRParser\r
+\r
+ Member variables inf_labase and inf_last were not initialized.\r
+ (See item #50.)\r
+\r
+#54. (Fixed in 1.33MR6) Brad Schick (schick@interacess.com)\r
+\r
+ Previously, the following constructs generated the same\r
+ code:\r
+\r
+ rule1 : (A B C)?\r
+ | something-else\r
+ ;\r
+\r
+ rule2 : (A B C)? ()\r
+ | something-else\r
+ ;\r
+\r
+ In all versions of pccts rule1 guesses (A B C) and then\r
+ consume all three tokens if the guess succeeds. In MR6\r
+ rule2 guesses (A B C) but consumes NONE of the tokens\r
+ when the guess succeeds because "()" matches epsilon.\r
+\r
+#53. (Explanation for 1.33MR6) What happens after an exception is caught ?\r
+\r
+ The Book is silent about what happens after an exception\r
+ is caught.\r
+\r
+ The following code fragment prints "Error Action" followed\r
+ by "Normal Action".\r
+\r
+ test : Word ex:Number <<printf("Normal Action\n");>>\r
+ exception[ex]\r
+ catch NoViableAlt:\r
+ <<printf("Error Action\n");>>\r
+ ;\r
+\r
+ The reason for "Normal Action" is that the normal flow of the\r
+ program after a user-written exception handler is to "drop through".\r
+ In the case of an exception handler for a rule this results in\r
+ the exection of a "return" statement. In the case of an\r
+ exception handler attached to an alternative, rule, or token\r
+ this is the code that would have executed had there been no\r
+ exception.\r
+\r
+ The user can achieve the desired result by using a "return"\r
+ statement.\r
+\r
+ test : Word ex:Number <<printf("Normal Action\n");>>\r
+ exception[ex]\r
+ catch NoViableAlt:\r
+ <<printf("Error Action\n"); return;>>\r
+ ;\r
+\r
+ The most powerful mechanism for recovery from parse errors\r
+ in pccts is syntactic predicates because they provide\r
+ backtracking. Exceptions allow "return", "break",\r
+ "consumeUntil(...)", "goto _handler", "goto _fail", and\r
+ changing the _signal value.\r
+\r
+#52. (Fixed in 1.33MR6) Exceptions without syntactic predicates\r
+\r
+ The following generates bad code in 1.33 if no syntactic\r
+ predicates are present in the grammar.\r
+\r
+ test : Word ex:Number <<printf("Normal Action\n");>>\r
+ exception[ex]\r
+ catch NoViableAlt:\r
+ <<printf("Error Action\n");>>\r
+\r
+ There is a reference to a guess variable. In C mode\r
+ this causes a compiler error. In C++ mode it generates\r
+ an extraneous check on member "guessing".\r
+\r
+ In MR6 correct code is generated for both C and C++ mode.\r
+\r
+#51. (Added to 1.33MR6) Exception operator "@" used without exceptions\r
+\r
+ In MR6 added a warning when the exception operator "@" is\r
+ used and no exception group is defined. This is probably\r
+ a case where "\@" or "@" is meant.\r
+\r
+#50. (Fixed in 1.33MR6) Gunnar Rxnning (gunnar@candleweb.no)\r
+ http://www.candleweb.no/~gunnar/\r
+\r
+ Routines zzsave_antlr_state and zzrestore_antlr_state don't\r
+ save and restore all the data needed when switching states.\r
+\r
+ Suggested patch applied to antlr.h and err.h for MR6.\r
+\r
+#49. (Fixed in 1.33MR6) Sinan Karasu (sinan@boeing.com)\r
+\r
+ Generated code failed to turn off guess mode when leaving a\r
+ (...)+ block which contained a guess block. The result was\r
+ an infinite loop. For example:\r
+\r
+ rule : (\r
+ (x)?\r
+ | y\r
+ )+\r
+\r
+ Suggested code fix implemented in MR6. Replaced\r
+\r
+ ... else if (zzcnt>1) break;\r
+\r
+ with:\r
+\r
+ C++ mode:\r
+ ... else if (zzcnt>1) {if (!zzrv) zzGUESS_DONE; break;};\r
+ C mode:\r
+ ... else if (zzcnt>1) {if (zzguessing) zzGUESS_DONE; break;};\r
+\r
+#48. (Fixed in 1.33MR6) Invalid exception element causes core\r
+\r
+ A label attached to an invalid construct can cause\r
+ pccts to crash while processing the exception associated\r
+ with the label. For example:\r
+\r
+ rule : t:(B C)\r
+ exception[t] catch MismatchedToken: <<printf(...);>>\r
+\r
+ Version MR6 generates the message:\r
+\r
+ reference in exception handler to undefined label 't'\r
+\r
+#47. (Fixed in 1.33MR6) Manuel Ornato\r
+\r
+ Under some circumstances involving a k >1 or ck >1\r
+ grammar and a loop block (i.e. (...)* ) pccts will\r
+ fail to detect a syntax error and loop indefinitely.\r
+ The problem did not exist in 1.20, but has existed\r
+ from 1.23 to the present.\r
+\r
+ Fixed in MR6.\r
+\r
+ ---------------------------------------------------\r
+ Complete test program\r
+ ---------------------------------------------------\r
+ #header<<\r
+ #include <stdio.h>\r
+ #include "charptr.h"\r
+ >>\r
+\r
+ <<\r
+ #include "charptr.c"\r
+ main ()\r
+ {\r
+ ANTLR(global(),stdin);\r
+ }\r
+ >>\r
+\r
+ #token "[\ \t]+" << zzskip(); >>\r
+ #token "[\n]" << zzline++; zzskip(); >>\r
+\r
+ #token B "b"\r
+ #token C "c"\r
+ #token D "d"\r
+ #token E "e"\r
+ #token LP "\("\r
+ #token RP "\)"\r
+\r
+ #token ANTLREOF "@"\r
+\r
+ global : (\r
+ (E liste)\r
+ | liste\r
+ | listed\r
+ ) ANTLREOF\r
+ ;\r
+\r
+ listeb : LP ( B ( B | C )* ) RP ;\r
+ listec : LP ( C ( B | C )* ) RP ;\r
+ listed : LP ( D ( B | C )* ) RP ;\r
+ liste : ( listeb | listec )* ;\r
+\r
+ ---------------------------------------------------\r
+ Sample data causing infinite loop\r
+ ---------------------------------------------------\r
+ e (d c)\r
+ ---------------------------------------------------\r
+\r
+#46. (Fixed in 1.33MR6) Robert Richter\r
+ (Robert.Richter@infotech.tu-chemnitz.de)\r
+\r
+ This item from the list of known problems was\r
+ fixed by item #18 (below).\r
+\r
+#45. (Fixed in 1.33MR6) Brad Schick (schick@interaccess.com)\r
+\r
+ The dependency scanner in VC++ mistakenly sees a\r
+ reference to an MPW #include file even though properly\r
+ #ifdef/#endif in config.h. The suggested workaround\r
+ has been implemented:\r
+\r
+ #ifdef MPW\r
+ .....\r
+ #define MPW_CursorCtl_Header <CursorCtl.h>\r
+ #include MPW_CursorCtl_Header\r
+ .....\r
+ #endif\r
+\r
+#44. (Fixed in 1.33MR6) cast malloc() to (char *) in charptr.c\r
+\r
+ Added (char *) cast for systems where malloc returns "void *".\r
+\r
+#43. (Added to 1.33MR6) Bruce Guenter (bruceg@qcc.sk.ca)\r
+\r
+ Add setLeft() and setUp methods to ASTDoublyLinkedBase\r
+ for symmetry with setRight() and setDown() methods.\r
+\r
+#42. (Fixed in 1.33MR6) Jeff Katcher (jkatcher@nortel.ca)\r
+\r
+ C++ style comment in antlr.c corrected.\r
+\r
+#41. (Added in 1.33MR6) antlr -stdout\r
+\r
+ Using "antlr -stdout ..." forces the text that would\r
+ normally go to the grammar.c or grammar.cpp file to\r
+ stdout.\r
+\r
+#40. (Added in 1.33MR6) antlr -tab to change tab stops\r
+\r
+ Using "antlr -tab number ..." changes the tab stops\r
+ for the grammar.c or grammar.cpp file. The number\r
+ must be between 0 and 8. Using 0 gives tab characters,\r
+ values between 1 and 8 give the appropriate number of\r
+ space characters.\r
+\r
+#39. (Fixed in 1.33MR5) Jan Mikkelsen <janm@zeta.org.au>\r
+\r
+ Commas in function prototype still not correct under\r
+ some circumstances. Suggested code fix installed.\r
+\r
+#38. (Fixed in 1.33MR5) ANTLRTokenBuffer constructor\r
+\r
+ Have ANTLRTokenBuffer ctor initialize member "parser" to null.\r
+\r
+#37. (Fixed in 1.33MR4) Bruce Guenter (bruceg@qcc.sk.ca)\r
+\r
+ In ANTLRParser::FAIL(int k,...) released memory pointed to by\r
+ f[i] (as well as f itself. Should only free f itself.\r
+\r
+#36. (Fixed in 1.33MR3) Cortland D. Starrett (cort@shay.ecn.purdue.edu)\r
+\r
+ Neglected to properly declare isDLGmaxToken() when fixing problem\r
+ reported by Andreas Magnusson.\r
+\r
+ Undo "_retv=NULL;" change which caused problems for return values\r
+ from rules whose return values weren't pointers.\r
+\r
+ Failed to create bin directory if it didn't exist.\r
+\r
+#35. (Fixed in 1.33MR2) Andreas Magnusson\r
+(Andreas.Magnusson@mailbox.swipnet.se)\r
+\r
+ Repair bug introduced by 1.33MR1 for #tokdefs. The original fix\r
+ placed "DLGmaxToken=9999" and "DLGminToken=0" in the TokenType enum\r
+ in order to fix a problem with an aggresive compiler assigning an 8\r
+ bit enum which might be too narrow. This caused #tokdefs to assume\r
+ that there were 9999 real tokens. The repair to the fix causes antlr to\r
+ ignore TokenTypes "DLGmaxToken" and "DLGminToken" in a #tokdefs file.\r
+\r
+#34. (Added to 1.33MR1) Add public DLGLexerBase::set_line(int newValue)\r
+\r
+ Previously there was no public function for changing the line\r
+ number maintained by the lexer.\r
+\r
+#33. (Fixed in 1.33MR1) Franklin Chen (chen@adi.com)\r
+\r
+ Accidental use of EXIT_FAILURE rather than PCCTS_EXIT_FAILURE\r
+ in pccts/h/AParser.cpp.\r
+\r
+#32. (Fixed in 1.33MR1) Franklin Chen (chen@adi.com)\r
+\r
+ In PCCTSAST.cpp lines 405 and 466: Change\r
+\r
+ free (t)\r
+ to\r
+ free ( (char *)t );\r
+\r
+ to match prototype.\r
+\r
+#31. (Added to 1.33MR1) Pointer to parser in ANTLRTokenBuffer\r
+ Pointer to parser in DLGLexerBase\r
+\r
+ The ANTLRTokenBuffer class now contains a pointer to the\r
+ parser which is using it. This is established by the\r
+ ANTLRParser constructor calling ANTLRTokenBuffer::\r
+ setParser(ANTLRParser *p).\r
+\r
+ When ANTLRTokenBuffer::setParser(ANTLRParser *p) is\r
+ called it saves the pointer to the parser and then\r
+ calls ANTLRTokenStream::setParser(ANTLRParser *p)\r
+ so that the lexer can also save a pointer to the\r
+ parser.\r
+\r
+ There is also a function getParser() in each class\r
+ with the obvious purpose.\r
+\r
+ It is possible that these functions will return NULL\r
+ under some circumstances (e.g. a non-DLG lexer is used).\r
+\r
+#30. (Added to 1.33MR1) function tokenName(int token) standard\r
+\r
+ The generated parser class now includes the\r
+ function:\r
+\r
+ static const ANTLRChar * tokenName(int token)\r
+\r
+ which returns a pointer to the "name" corresponding\r
+ to the token.\r
+\r
+ The base class (ANTLRParser) always includes the\r
+ member function:\r
+\r
+ const ANTLRChar * parserTokenName(int token)\r
+\r
+ which can be accessed by objects which have a pointer\r
+ to an ANTLRParser, but do not know the name of the\r
+ parser class (e.g. ANTLRTokenBuffer and DLGLexerBase).\r
+\r
+#29. (Added to 1.33MR1) Debugging DLG lexers\r
+\r
+ If the pre-processor symbol DEBUG_LEXER is defined\r
+ then DLexerBase will include code for printing out\r
+ key information about tokens which are recognized.\r
+\r
+ The debug feature of the lexer is controlled by:\r
+\r
+ int previousDebugValue=lexer.debugLexer(newValue);\r
+\r
+ a value of 0 disables output\r
+ a value of 1 enables output\r
+\r
+ Even if the lexer debug code is compiled into DLexerBase\r
+ it must be enabled before any output is generated. For\r
+ example:\r
+\r
+ DLGFileInput in(stdin);\r
+ MyDLG lexer(&in,2000);\r
+\r
+ lexer.setToken(&aToken);\r
+\r
+ #if DEBUG_LEXER\r
+ lexer.debugLexer(1); // enable debug information\r
+ #endif\r
+\r
+#28. (Added to 1.33MR1) More control over DLG header\r
+\r
+ Version 1.33MR1 adds the following directives to PCCTS\r
+ for C++ mode:\r
+\r
+ #lexprefix <<source code>>\r
+\r
+ Adds source code to the DLGLexer.h file\r
+ after the #include "DLexerBase.h" but\r
+ before the start of the class definition.\r
+\r
+ #lexmember <<source code>>\r
+\r
+ Adds source code to the DLGLexer.h file\r
+ as part of the DLGLexer class body. It\r
+ appears immediately after the start of\r
+ the class and a "public: statement.\r
+\r
+#27. (Fixed in 1.33MR1) Comments in DLG actions\r
+\r
+ Previously, DLG would not recognize comments as a special case.\r
+ Thus, ">>" in the comments would cause errors. This is fixed.\r
+\r
+#26. (Fixed in 1.33MR1) Removed static variables from error routines\r
+\r
+ Previously, the existence of statically allocated variables\r
+ in some of the parser's member functions posed a danger when\r
+ there was more than one parser active.\r
+\r
+ Replaced with dynamically allocated/freed variables in 1.33MR1.\r
+\r
+#25. (Fixed in 1.33MR1) Use of string literals in semantic predicates\r
+\r
+ Previously, it was not possible to place a string literal in\r
+ a semantic predicate because it was not properly "stringized"\r
+ for the report of a failed predicate.\r
+\r
+#24. (Fixed in 1.33MR1) Continuation lines for semantic predicates\r
+\r
+ Previously, it was not possible to continue semantic\r
+ predicates across a line because it was not properly\r
+ "stringized" for the report of a failed predicate.\r
+\r
+ rule : <<ifXYZ()>>?[ a very\r
+ long statement ]\r
+\r
+#23. (Fixed in 1.33MR1) {...} envelope for failed semantic predicates\r
+\r
+ Previously, there was a code generation error for failed\r
+ semantic predicates:\r
+\r
+ rule : <<xyz()>>?[ stmt1; stmt2; ]\r
+\r
+ which generated code which resembled:\r
+\r
+ if (! xyz()) stmt1; stmt2;\r
+\r
+ It now puts the statements in a {...} envelope:\r
+\r
+ if (! xyz()) { stmt1; stmt2; };\r
+\r
+#22. (Fixed in 1.33MR1) Continuation of #token across lines using "\"\r
+\r
+ Previously, it was not possible to continue a #token regular\r
+ expression across a line. The trailing "\" and newline caused\r
+ a newline to be inserted into the regular expression by DLG.\r
+\r
+ Fixed in 1.33MR1.\r
+\r
+#21. (Fixed in 1.33MR1) Use of ">>" (right shift operator in DLG actions\r
+\r
+ It is now possible to use the C++ right shift operator ">>"\r
+ in DLG actions by using the normal escapes:\r
+\r
+ #token "shift-right" << value=value \>\> 1;>>\r
+\r
+#20. (Version 1.33/19-Jan-97 Karl Eccleson <karle@microrobotics.co.uk>\r
+ P.A. Keller (P.A.Keller@bath.ac.uk)\r
+\r
+ There is a problem due to using exceptions with the -gh option.\r
+\r
+ Suggested fix now in 1.33MR1.\r
+\r
+#19. (Fixed in 1.33MR1) Tom Piscotti and John Lilley\r
+\r
+ There were problems suppressing messages to stdin and stdout\r
+ when running in a window environment because some functions\r
+ which uses fprint were not virtual.\r
+\r
+ Suggested change now in 1.33MR1.\r
+\r
+ I believe all functions containing error messages (excluding those\r
+ indicating internal inconsistency) have been placed in functions\r
+ which are virtual.\r
+\r
+#18. (Version 1.33/ 22-Nov-96) John Bair (jbair@iftime.com)\r
+\r
+ Under some combination of options a required "return _retv" is\r
+ not generated.\r
+\r
+ Suggested fix now in 1.33MR1.\r
+\r
+#17. (Version 1.33/3-Sep-96) Ron House (house@helios.usq.edu.au)\r
+\r
+ The routine ASTBase::predorder_action omits two "tree->"\r
+ prefixes, which results in the preorder_action belonging\r
+ to the wrong node to be invoked.\r
+\r
+ Suggested fix now in 1.33MR1.\r
+\r
+#16. (Version 1.33/7-Jun-96) Eli Sternheim <eli@interhdl.com>\r
+\r
+ Routine consumeUntilToken() does not check for end-of-file\r
+ condition.\r
+\r
+ Suggested fix now in 1.33MR1.\r
+\r
+#15. (Version 1.33/8 Apr 96) Asgeir Olafsson <olafsson@cstar.ac.com>\r
+\r
+ Problem with tree duplication of doubly linked ASTs in ASTBase.cpp.\r
+\r
+ Suggested fix now in 1.33MR1.\r
+\r
+#14. (Version 1.33/28-Feb-96) Andreas.Magnusson@mailbox.swipnet.se\r
+\r
+ Problem with definition of operator = (const ANTLRTokenPtr rhs).\r
+\r
+ Suggested fix now in 1.33MR1.\r
+\r
+#13. (Version 1.33/13-Feb-96) Franklin Chen (chen@adi.com)\r
+\r
+ Sun C++ Compiler 3.0.1 can't compile testcpp/1 due to goto in\r
+ block with destructors.\r
+\r
+ Apparently fixed. Can't locate "goto".\r
+\r
+#12. (Version 1.33/10-Nov-95) Minor problems with 1.33 code\r
+\r
+ The following items have been fixed in 1.33MR1:\r
+\r
+ 1. pccts/antlr/main.c line 142\r
+\r
+ "void" appears in classic C code\r
+\r
+ 2. no makefile in support/genmk\r
+\r
+ 3. EXIT_FAILURE/_SUCCESS instead of PCCTS_EXIT_FAILURE/_SUCCESS\r
+\r
+ pccts/h/PCCTSAST.cpp\r
+ pccts/h/DLexerBase.cpp\r
+ pccts/testcpp/6/test.g\r
+\r
+ 4. use of "signed int" isn't accepted by AT&T cfront\r
+\r
+ pccts/h/PCCTSAST.h line 42\r
+\r
+ 5. in call to ANTLRParser::FAIL the var arg err_k is passed as\r
+ "int" but is declared "unsigned int".\r
+\r
+ 6. I believe that a failed validation predicate still does not\r
+ get put in a "{...}" envelope, despite the release notes.\r
+\r
+ 7. The #token ">>" appearing in the DLG grammar description\r
+ causes DLG to generate the string literal "\>\>" which\r
+ is non-conforming and will cause some compilers to\r
+ complain (scan.c function act10 line 143 of source code).\r
+\r
+#11. (Version 1.32b6) Dave Kuhlman (dkuhlman@netcom.com)\r
+\r
+ Problem with file close in gen.c. Already fixed in 1.33.\r
+\r
+#10. (Version 1.32b6/29-Aug-95)\r
+\r
+ pccts/antlr/main.c contains a C++ style comments on lines 149\r
+ and 176 which causes problems for most C compilers.\r
+\r
+ Already fixed in 1.33.\r
+\r
+#9. (Version 1.32b4/14-Mar-95) dlgauto.h #include "config.h"\r
+\r
+ The file pccts/h/dlgauto.h should probably contain a #include\r
+ "config.h" as it uses the #define symbol __USE_PROTOS.\r
+\r
+ Added to 1.33MR1.\r
+\r
+#8. (Version 1.32b4/6-Mar-95) Michael T. Richter (mtr@igs.net)\r
+\r
+ In C++ output mode anonymous tokens from in-line regular expressions\r
+ can create enum values which are too wide for the datatype of the enum\r
+ assigned by the C++ compiler.\r
+\r
+ Fixed in 1.33MR1.\r
+\r
+#7. (Version 1.32b4/6-Mar-95) C++ does not imply __STDC__\r
+\r
+ In err.h the combination of # directives assumes that a C++\r
+ compiler has __STDC__ defined. This is not necessarily true.\r
+\r
+ This problem also appears in the use of __USE_PROTOS which\r
+ is appropriate for both Standard C and C++ in antlr/gen.c\r
+ and antlr/lex.c\r
+\r
+ Fixed in 1.33MR1.\r
+\r
+#6. (Version 1.32 ?/15-Feb-95) Name conflict for "TokenType"\r
+\r
+ Already fixed in 1.33.\r
+\r
+#5. (23-Jan-95) Douglas_Cuthbertson.JTIDS@jtids_qmail.hanscom.af.mil\r
+\r
+ The fail action following a semantic predicate is not enclosed in\r
+ "{...}". This can lead to problems when the fail action contains\r
+ more than one statement.\r
+\r
+ Fixed in 1.33MR1.\r
+\r
+#4 . (Version 1.33/31-Mar-96) jlilley@empathy.com (John Lilley)\r
+\r
+ Put briefly, a semantic predicate ought to abort a guess if it fails.\r
+\r
+ Correction suggested by J. Lilley has been added to 1.33MR1.\r
+\r
+#3 . (Version 1.33) P.A.Keller@bath.ac.uk\r
+\r
+ Extra commas are placed in the K&R style argument list for rules\r
+ when using both exceptions and ASTs.\r
+\r
+ Fixed in 1.33MR1.\r
+\r
+#2. (Version 1.32b6/2-Oct-95) Brad Schick <schick@interaccess.com>\r
+\r
+ Construct #[] generates zzastnew() in C++ mode.\r
+\r
+ Already fixed in 1.33.\r
+\r
+#1. (Version 1.33) Bob Bailey (robert@oakhill.sps.mot.com)\r
+\r
+ Previously, config.h assumed that all PC systems required\r
+ "short" file names. The user can now override that\r
+ assumption with "#define LONGFILENAMES".\r
+\r
+ Added to 1.33MR1.\r
projects / mirror_edk2.git / blobdiff