1 ======================================================================
5 A QUICK overview of changes from 1.33 in reverse order
7 A summary of additions rather than bug fixes and minor code changes.
9 Numbers refer to items in CHANGES_FROM_133*.TXT
10 which may contain additional information.
14 The software and these notes are provided "as is". They may include
15 typographical or technical errors and their authors disclaims all
16 liability of any kind or nature for damages due to error, fault,
17 defect, or deficiency regardless of cause. All warranties of any
18 kind, either express or implied, including, but not limited to, the
19 implied warranties of merchantability and fitness for a particular
20 purpose are disclaimed.
22 ======================================================================
24 #258. You can specify a user-defined base class for your parser
26 The base class must constructor must have a signature similar to
29 #253. Generation of block preamble (-preamble and -preamble_first)
31 The antlr option -preamble causes antlr to insert the code
32 BLOCK_PREAMBLE at the start of each rule and block.
34 The antlr option -preamble_first is similar, but inserts the
35 code BLOCK_PREAMBLE_FIRST(PreambleFirst_123) where the symbol
36 PreambleFirst_123 is equivalent to the first set defined by
37 the #FirstSetSymbol described in Item #248.
39 #248. Generate symbol for first set of an alternative
41 rr : #FirstSetSymbol(rr_FirstSet) ( Foo | Bar ) ;
43 #216. Defer token fetch for C++ mode
45 When the ANTLRParser class is built with the pre-processor option
46 ZZDEFER_FETCH defined, the fetch of new tokens by consume() is deferred
47 until LA(i) or LT(i) is called.
49 #215. Use reset() to reset DLGLexerBase
50 #188. Added pccts/h/DLG_stream_input.h
51 #180. Added ANTLRParser::getEofToken()
52 #173. -glms for Microsoft style filenames with -gl
53 #170. Suppression for predicates with lookahead depth >1
55 Consider the following grammar with -ck 2 and the predicate in rule
65 a : (A B)? => <<p(LATEXT(2))>>? A B C
71 Normally, the predicate would be hoisted into rule r1 in order to
72 determine whether to call rule "ab". However it should *not* be
73 hoisted because, even if p is false, there is a valid alternative
74 in rule b. With "-mrhoistk on" the predicate will be suppressed.
76 If "-info p" command line option is present the following information
77 will appear in the generated code:
82 Part (or all) of predicate with depth > 1 suppressed by alternative
85 pred << p(LATEXT(2))>>?
86 depth=k=2 ("=>" guard) rule a line 8 t1.g
92 The token sequence which is suppressed: ( A B )
93 The sequence of references which generate that sequence of tokens:
95 1 to ab r1/1 line 1 t1.g
97 3 to b ab/2 line 5 t1.g
99 5 #token A b/1 line 11 t1.g
100 6 #token B b/1 line 11 t1.g
104 A slightly more complicated example:
113 a : (A B)? => <<p(LATEXT(2))>>? (A B | D E)
116 b : <<q(LATEXT(2))>>? D E
120 In this case, the sequence (D E) in rule "a" which lies behind
121 the guard is used to suppress the predicate with context (D E)
124 while ( (LA(1)==A || LA(1)==D)
127 Part (or all) of predicate with depth > 1 suppressed by alternative
130 pred << q(LATEXT(2))>>?
131 depth=k=2 rule b line 11 t2.g
137 The token sequence which is suppressed: ( D E )
138 The sequence of references which generate that sequence of tokens:
140 1 to ab r1/1 line 1 t2.g
141 2 ab ab/1 line 4 t2.g
142 3 to a ab/1 line 4 t2.g
144 5 #token D a/1 line 8 t2.g
145 6 #token E a/1 line 8 t2.g
151 pred << p(LATEXT(2))>>?
152 depth=k=2 ("=>" guard) rule a line 8 t2.g
160 (! ( LA(1)==A && LA(2)==B ) || p(LATEXT(2)) ) {
164 #165. (Changed in MR13) option -newAST
166 To create ASTs from an ANTLRTokenPtr antlr usually calls
167 "new AST(ANTLRTokenPtr)". This option generates a call
168 to "newAST(ANTLRTokenPtr)" instead. This allows a user
169 to define a parser member function to create an AST object.
171 #161. (Changed in MR13) Switch -gxt inhibits generation of tokens.h
173 #158. (Changed in MR13) #header causes problem for pre-processors
175 A user who runs the C pre-processor on antlr source suggested
176 that another syntax be allowed. With MR13 such directives
177 such as #header, #pragma, etc. may be written as "\#header",
178 "\#pragma", etc. For escaping pre-processor directives inside
179 a #header use something like the following:
186 #155. (Changed in MR13) Context behind predicates can suppress
188 With -mrhoist enabled the context behind a guarded predicate can
189 be used to suppress other predicates. Consider the following grammar:
196 rp : <<p LATEXT(1)>>? B ;
197 rq : (A)? => <<q LATEXT(1)>>? (A|B);
199 In earlier versions both predicates "p" and "q" would be hoisted into
200 rule r0. With MR12c predicate p is suppressed because the context which
201 follows predicate q includes "B" which can "cover" predicate "p". In
202 other words, in trying to decide in r0 whether to call r1, it doesn't
203 really matter whether p is false or true because, either way, there is
204 a valid choice within r1.
206 #154. (Changed in MR13) Making hoist suppression explicit using <<nohoist>>
208 A common error, even among experienced pccts users, is to code
209 an init-action to inhibit hoisting rather than a leading action.
210 An init-action does not inhibit hoisting.
216 This is what was meant:
218 rule1 : <<;>> <<;>> rule2
220 With MR13, the user can code:
222 rule1 : <<;>> <<nohoist>> rule2
224 The following will give an error message:
226 rule1 : <<nohoist>> rule2
228 If the <<nohoist>> appears as an init-action rather than a leading
229 action an error message is issued. The meaning of an init-action
230 containing "nohoist" is unclear: does it apply to just one
231 alternative or to all alternatives ?
233 #151a. Addition of ANTLRParser::getLexer(), ANTLRTokenStream::getLexer()
235 You must manually cast the ANTLRTokenStream to your program's
236 lexer class. Because the name of the lexer's class is not fixed.
237 Thus it is impossible to incorporate it into the DLGLexerBase
240 #151b.(Changed in MR12) ParserBlackBox member getLexer()
242 #150. (Changed in MR12) syntaxErrCount and lexErrCount now public
244 #149. (Changed in MR12) antlr option -info o (letter o for orphan)
246 If there is more than one rule which is not referenced by any
247 other rule then all such rules are listed. This is useful for
248 alerting one to rules which are not used, but which can still
249 contribute to ambiguity.
251 #148. (Changed in MR11) #token names appearing in zztokens,token_tbl
255 #token Plus ("+") "\+"
257 #token COM ("comment begin") "/\*"
259 The string in parenthesis will be used in syntax error messages.
261 #146. (Changed in MR11) Option -treport for locating "difficult" alts
263 It can be difficult to determine which alternatives are causing
264 pccts to work hard to resolve an ambiguity. In some cases the
265 ambiguity is successfully resolved after much CPU time so there
266 is no message at all.
268 A rough measure of the amount of work being peformed which is
269 independent of the CPU speed and system load is the number of
270 tnodes created. Using "-info t" gives information about the
271 total number of tnodes created and the peak number of tnodes.
273 Tree Nodes: peak 1300k created 1416k lost 0
275 It also puts in the generated C or C++ file the number of tnodes
276 created for a rule (at the end of the rule). However this
277 information is not sufficient to locate the alternatives within
278 a rule which are causing the creation of tnodes.
282 antlr -treport 100000 ....
284 causes antlr to list on stdout any alternatives which require the
285 creation of more than 100,000 tnodes, along with the lookahead sets
286 for those alternatives.
288 The following is a trivial case from the ansi.g grammar which shows
289 the format of the report. This report might be of more interest
290 in cases where 1,000,000 tuples were created to resolve the ambiguity.
292 -------------------------------------------------------------------------
293 There were 0 tuples whose ambiguity could not be resolved
295 There were 157 tnodes created to resolve ambiguity between:
297 Choice 1: statement/2 line 475 file ansi.g
298 Choice 2: statement/3 line 476 file ansi.g
300 Intersection of lookahead[1] sets:
304 Intersection of lookahead[2] sets:
306 LPARENTHESIS COLON AMPERSAND MINUS
307 STAR PLUSPLUS MINUSMINUS ONESCOMPLEMENT
308 NOT SIZEOF OCTALINT DECIMALINT
309 HEXADECIMALINT FLOATONE FLOATTWO IDENTIFIER
311 -------------------------------------------------------------------------
313 #143. (Changed in MR11) Optional ";" at end of #token statement
325 #token X "x" <<lexical action>>
327 #142. (Changed in MR11) class BufFileInput subclass of DLGInputStream
329 Alexey Demakov (demakov@kazbek.ispras.ru) has supplied class
330 BufFileInput derived from DLGInputStream which provides a
331 function lookahead(char *string) to test characters in the
332 input stream more than one character ahead.
333 The class is located in pccts/h/BufFileInput.* of the kit.
335 #140. #pred to define predicates
337 +---------------------------------------------------+
338 | Note: Assume "-prc on" for this entire discussion |
339 +---------------------------------------------------+
341 A problem with predicates is that each one is regarded as
342 unique and capable of disambiguating cases where two
343 alternatives have identical lookahead. For example:
345 rule : <<pred(LATEXT(1))>>? A
346 | <<pred(LATEXT(1))>>? A
349 will not cause any error messages or warnings to be issued
350 by earlier versions of pccts. To compare the text of the
351 predicates is an incomplete solution.
353 In 1.33MR11 I am introducing the #pred statement in order to
354 solve some problems with predicates. The #pred statement allows
355 one to give a symbolic name to a "predicate literal" or a
356 "predicate expression" in order to refer to it in other predicate
357 expressions or in the rules of the grammar.
359 The predicate literal associated with a predicate symbol is C
360 or C++ code which can be used to test the condition. A
361 predicate expression defines a predicate symbol in terms of other
362 predicate symbols using "!", "&&", and "||". A predicate symbol
363 can be defined in terms of a predicate literal, a predicate
364 expression, or *both*.
366 When a predicate symbol is defined with both a predicate literal
367 and a predicate expression, the predicate literal is used to generate
368 code, but the predicate expression is used to check for two
369 alternatives with identical predicates in both alternatives.
371 Here are some examples of #pred statements:
373 #pred IsLabel <<isLabel(LATEXT(1))>>?
374 #pred IsLocalVar <<isLocalVar(LATEXT(1))>>?
375 #pred IsGlobalVar <<isGlobalVar(LATEXT(1)>>?
376 #pred IsVar <<isVar(LATEXT(1))>>? IsLocalVar || IsGlobalVar
377 #pred IsScoped <<isScoped(LATEXT(1))>>? IsLabel || IsLocalVar
379 I hope that the use of EBNF notation to describe the syntax of the
380 #pred statement will not cause problems for my readers (joke).
382 predStatement : "#pred"
385 "<<predicate_literal>>?"
386 | "<<predicate_literal>>?" predOrExpr
391 predOrExpr : predAndExpr ( "||" predAndExpr ) * ;
393 predAndExpr : predPrimary ( "&&" predPrimary ) * ;
395 predPrimary : CapitalizedName
400 What is the purpose of this nonsense ?
402 To understand how predicate symbols help, you need to realize that
403 predicate symbols are used in two different ways with two different
406 a. Allow simplification of predicates which have been combined
407 during predicate hoisting.
409 b. Allow recognition of identical predicates which can't disambiguate
410 alternatives with common lookahead.
412 First we will discuss goal (a). Consider the following rule:
423 rule2: <<isX(LATEXT(1))>>? ID ;
424 rule3: <<!isX(LATEXT(1)>>? ID ;
426 When the predicates in rule2 and rule3 are combined by hoisting
427 to create a prediction expression for rule1 the result is:
430 && ( isX(LATEXT(1) || !isX(LATEXT(1) ) ) { rule1(); ...
432 This is inefficient, but more importantly, can lead to false
433 assumptions that the predicate expression distinguishes the rule1
434 alternative with some other alternative with lookahead ID. In
437 #pred IsX <<isX(LATEXT(1))>>?
442 rule3: <<!IsX>>? ID ;
444 During hoisting MR11 recognizes this as a special case and
445 eliminates the predicates. The result is a prediction
446 expression like the following:
448 if ( LA(1)==ID ) { rule1(); ...
450 Please note that the following cases which appear to be equivalent
451 *cannot* be simplified by MR11 during hoisting because the hoisting
452 logic only checks for a "!" in the predicate action, not in the
453 predicate expression for a predicate symbol.
455 *Not* equivalent and is not simplified during hoisting:
457 #pred IsX <<isX(LATEXT(1))>>?
458 #pred NotX <<!isX(LATEXT(1))>>?
461 rule3: <<NotX>>? ID ;
463 *Not* equivalent and is not simplified during hoisting:
465 #pred IsX <<isX(LATEXT(1))>>?
469 rule3: <<NotX>>? ID ;
471 Now we will discuss goal (b).
473 When antlr discovers that there is a lookahead ambiguity between
474 two alternatives it attempts to resolve the ambiguity by searching
475 for predicates in both alternatives. In the past any predicate
476 would do, even if the same one appeared in both alternatives:
478 rule: <<p(LATEXT(1))>>? X
479 | <<p(LATEXT(1))>>? X
482 The #pred statement is a start towards solving this problem.
483 During ambiguity resolution (*not* predicate hoisting) the
484 predicates for the two alternatives are expanded and compared.
485 Consider the following example:
487 #pred Upper <<isUpper(LATEXT(1))>>?
488 #pred Lower <<isLower(LATEXT(1))>>?
489 #pred Alpha <<isAlpha(LATEXT(1))>>? Upper || Lower
501 rule2: <<Upper>>? ID;
502 rule3: <<Lower>>? ID;
504 The definition of #pred Alpha expresses:
506 a. to test the predicate use the C code "isAlpha(LATEXT(1))"
508 b. to analyze the predicate use the information that
509 Alpha is equivalent to the union of Upper and Lower,
511 During ambiguity resolution the definition of Alpha is expanded
512 into "Upper || Lower" and compared with the predicate in the other
513 alternative, which is also "Upper || Lower". Because they are
514 identical MR11 will report a problem.
516 -------------------------------------------------------------------------
517 t10.g, line 5: warning: the predicates used to disambiguate rule rule0
518 (file t10.g alt 1 line 5 and alt 2 line 6)
519 are identical when compared without context and may have no
520 resolving power for some lookahead sequences.
521 -------------------------------------------------------------------------
523 If you use the "-info p" option the output file will contain:
525 +----------------------------------------------------------------------+
528 |The following predicates are identical when compared without |
529 | lookahead context information. For some ambiguous lookahead |
530 | sequences they may not have any power to resolve the ambiguity. |
532 |Choice 1: rule0/1 alt 1 line 5 file t10.g |
534 | The original predicate for choice 1 with available context |
540 | depth=k=1 rule rule2 line 14 t10.g |
545 | depth=k=1 rule rule3 line 15 t10.g |
549 | The predicate for choice 1 after expansion (but without context |
554 | pred << isUpper(LATEXT(1))>>? |
555 | depth=k=1 rule line 1 t10.g |
557 | pred << isLower(LATEXT(1))>>? |
558 | depth=k=1 rule line 2 t10.g |
561 |Choice 2: rule0/2 alt 2 line 6 file t10.g |
563 | The original predicate for choice 2 with available context |
567 | depth=k=1 rule rule0 line 6 t10.g |
571 | The predicate for choice 2 after expansion (but without context |
576 | pred << isUpper(LATEXT(1))>>? |
577 | depth=k=1 rule line 1 t10.g |
579 | pred << isLower(LATEXT(1))>>? |
580 | depth=k=1 rule line 2 t10.g |
584 +----------------------------------------------------------------------+
586 The comparison of the predicates for the two alternatives takes
587 place without context information, which means that in some cases
588 the predicates will be considered identical even though they operate
589 on disjoint lookahead sets. Consider:
597 Because the comparison of predicates takes place without context
598 these will be considered identical. The reason for comparing
599 without context is that otherwise it would be necessary to re-evaluate
600 the entire predicate expression for each possible lookahead sequence.
601 This would require more code to be written and more CPU time during
602 grammar analysis, and it is not yet clear whether anyone will even make
603 use of the new #pred facility.
605 A temporary workaround might be to use different #pred statements
606 for predicates you know have different context. This would avoid
609 The above example might be termed a "false positive". Comparison
610 without context will also lead to "false negatives". Consider the
624 The predicate used for alt 2 of rule1 is (Alpha || Beta). This
625 appears to be different than the predicate Alpha used for alt1.
626 However, the context of Beta is B. Thus when the lookahead is A
627 Beta will have no resolving power and Alpha will be used for both
628 alternatives. Using the same predicate for both alternatives isn't
629 very helpful, but this will not be detected with 1.33MR11.
631 To properly handle this the predicate expression would have to be
632 evaluated for each distinct lookahead context.
634 To determine whether two predicate expressions are identical is
635 difficult. The routine may fail to identify identical predicates.
637 The #pred feature also compares predicates to see if a choice between
638 alternatives which is resolved by a predicate which makes the second
639 choice unreachable. Consider the following example:
641 #pred A <<A(LATEXT(1)>>?
642 #pred B <<B(LATEXT(1)>>?
653 ----------------------------------------------------------------------------
654 t11.g, line 5: warning: the predicate used to disambiguate the
655 first choice of rule r
656 (file t11.g alt 1 line 5 and alt 2 line 6)
657 appears to "cover" the second predicate when compared without context.
658 The second predicate may have no resolving power for some lookahead
660 ----------------------------------------------------------------------------
662 #132. (Changed in 1.33MR11) Recognition of identical predicates in alts
664 Prior to 1.33MR11, there would be no ambiguity warning when the
665 very same predicate was used to disambiguate both alternatives:
671 ref : <<pred(LATEXT(1)>>? A
673 In 1.33MR11 this will cause the warning:
675 warning: the predicates used to disambiguate rule test
676 (file v98.g alt 1 line 1 and alt 2 line 2)
677 are identical and have no resolving power
679 ----------------- Note -----------------
681 This is different than the following case
683 test: <<pred(LATEXT(1))>>? A B
684 | <<pred(LATEXT(1)>>? A C
687 In this case there are two distinct predicates
688 which have exactly the same text. In the first
689 example there are two references to the same
690 predicate. The problem represented by this
691 grammar will be addressed later.
694 #127. (Changed in 1.33MR11)
696 Count Syntax Errors Count DLG Errors
697 ------------------- ----------------
699 C++ mode ANTLRParser:: DLGLexerBase::
700 syntaxErrCount lexErrCount
701 C mode zzSyntaxErrCount zzLexErrCount
703 The C mode variables are global and initialized to 0.
704 They are *not* reset to 0 automatically when antlr is
707 The C++ mode variables are public. They are initialized
708 to 0 by the constructors. They are *not* reset to 0 by the
709 ANTLRParser::init() method.
711 Suggested by Reinier van den Born (reinier@vnet.ibm.com).
713 #126. (Changed in 1.33MR11) Addition of #first <<...>>
715 The #first <<...>> inserts the specified text in the output
716 files before any other #include statements required by pccts.
717 The only things before the #first text are comments and
718 a #define ANTLR_VERSION.
720 Requested by and Esa Pulkkinen (esap@cs.tut.fi) and Alexin
721 Zoltan (alexin@inf.u-szeged.hu).
723 #124. A Note on the New "&&" Style Guarded Predicates
725 I've been asked several times, "What is the difference between
726 the old "=>" style guard predicates and the new style "&&" guard
727 predicates, and how do you choose one over the other" ?
729 The main difference is that the "=>" does not apply the
730 predicate if the context guard doesn't match, whereas
731 the && form always does. What is the significance ?
733 If you have a predicate which is not on the "leading edge"
734 it is cannot be hoisted. Suppose you need a predicate that
735 looks at LA(2). You must introduce it manually. The
743 typeName : <<isTypeName(LATEXT(1))>>? ID
747 The problem is that isTypeName() isn't on the leading edge
748 of typeName, so it won't be hoisted into castExpr to help
749 make a decision on which production to choose.
751 The *first* attempt to fix it is this:
754 <<isTypeName(LATEXT(2))>>?
759 Unfortunately, this won't work because it ignores
760 the problem of STRUCT. The solution is to apply
761 isTypeName() in castExpr if LA(2) is an ID and
762 don't apply it when LA(2) is STRUCT:
765 (LP ID)? => <<isTypeName(LATEXT(2))>>?
770 In conclusion, the "=>" style guarded predicate is
773 a. the tokens required for the predicate
774 are not on the leading edge
775 b. there are alternatives in the expression
776 selected by the predicate for which the
777 predicate is inappropriate
779 If (b) were false, then one could use a simple
780 predicate (assuming "-prc on"):
783 <<isTypeName(LATEXT(2))>>?
788 typeName : <<isTypeName(LATEXT(1))>>? ID
791 So, when do you use the "&&" style guarded predicate ?
793 The new-style "&&" predicate should always be used with
794 predicate context. The context guard is in ADDITION to
795 the automatically computed context. Thus it useful for
796 predicates which depend on the token type for reasons
799 The following example is contributed by Reinier van den Born
800 (reinier@vnet.ibm.com).
802 +-------------------------------------------------------------------------+
803 | This grammar has two ways to call functions: |
805 | - a "standard" call syntax with parens and comma separated args |
806 | - a shell command like syntax (no parens and spacing separated args) |
808 | The former also allows a variable to hold the name of the function, |
809 | the latter can also be used to call external commands. |
811 | The grammar (simplified) looks like this: |
813 | fun_call : ID "(" { expr ("," expr)* } ")" |
814 | /* ID is function name */ |
815 | | "@" ID "(" { expr ("," expr)* } ")" |
816 | /* ID is var containing fun name */ |
819 | command : ID expr* /* ID is function name */ |
820 | | path expr* /* path is external command name */ |
823 | path : ID /* left out slashes and such */ |
824 | | "@" ID /* ID is environment var */ |
834 | Obviously the call is wildly ambiguous. This is more or less how this |
835 | is to be resolved: |
837 | A call begins with an ID or an @ followed by an ID. |
839 | If it is an ID and if it is an ext. command name -> command |
840 | if followed by a paren -> fun_call |
841 | otherwise -> command |
843 | If it is an @ and if the ID is a var name -> fun_call |
844 | otherwise -> command |
846 | One can implement these rules quite neatly using && predicates: |
848 | call : ("@" ID)? && <<isVarName(LT(2))>>? fun_call |
849 | | (ID)? && <<isExtCmdName>>? command |
850 | | (ID "(")? fun_call |
854 | This can be done better, so it is not an ideal example, but it |
855 | conveys the principle. |
856 +-------------------------------------------------------------------------+
858 #122. (Changed in 1.33MR11) Member functions to reset DLG in C++ mode
860 void DLGFileReset(FILE *f) { input = f; found_eof = 0; }
861 void DLGStringReset(DLGChar *s) { input = s; p = &input[0]; }
863 Supplied by R.A. Nelson (cowboy@VNET.IBM.COM)
865 #119. (Changed in 1.33MR11) Ambiguity aid for grammars
867 The user can ask for additional information on ambiguities reported
868 by antlr to stdout. At the moment, only one ambiguity report can
869 be created in an antlr run.
871 This feature is enabled using the "-aa" (Ambiguity Aid) option.
873 The following options control the reporting of ambiguities:
875 -aa ruleName Selects reporting by name of rule
876 -aa lineNumber Selects reporting by line number
877 (file name not compared)
879 -aam Selects "multiple" reporting for a token
880 in the intersection set of the
883 For instance, the token ID may appear dozens
884 of times in various paths as the program
885 explores the rules which are reachable from
886 the point of an ambiguity. With option -aam
887 every possible path the search program
888 encounters is reported.
890 Without -aam only the first encounter is
891 reported. This may result in incomplete
892 information, but the information may be
893 sufficient and much shorter.
895 -aad depth Selects the depth of the search.
896 The default value is 1.
898 The number of paths to be searched, and the
899 size of the report can grow geometrically
900 with the -ck value if a full search for all
901 contributions to the source of the ambiguity
904 The depth represents the number of tokens
905 in the lookahead set which are matched against
906 the set of ambiguous tokens. A depth of 1
907 means that the search stops when a lookahead
908 sequence of just one token is matched.
910 A k=1 ck=6 grammar might generate 5,000 items
911 in a report if a full depth 6 search is made
912 with the Ambiguity Aid. The source of the
913 problem may be in the first token and obscured
914 by the volume of data - I hesitate to call
917 When the user selects a depth > 1, the search
918 is first performed at depth=1 for both
919 alternatives, then depth=2 for both alternatives,
922 Sample output for rule grammar in antlr.g itself:
924 +---------------------------------------------------------------------+
927 | Choice 1: grammar/70 line 632 file a.g |
928 | Choice 2: grammar/82 line 644 file a.g |
930 | Intersection of lookahead[1] sets: |
932 | "\}" "class" "#errclass" "#tokclass" |
934 | Choice:1 Depth:1 Group:1 ("#errclass") |
935 | 1 in (...)* block grammar/70 line 632 a.g |
936 | 2 to error grammar/73 line 635 a.g |
937 | 3 error error/1 line 894 a.g |
938 | 4 #token "#errclass" error/2 line 895 a.g |
940 | Choice:1 Depth:1 Group:2 ("#tokclass") |
941 | 2 to tclass grammar/74 line 636 a.g |
942 | 3 tclass tclass/1 line 937 a.g |
943 | 4 #token "#tokclass" tclass/2 line 938 a.g |
945 | Choice:1 Depth:1 Group:3 ("class") |
946 | 2 to class_def grammar/75 line 637 a.g |
947 | 3 class_def class_def/1 line 669 a.g |
948 | 4 #token "class" class_def/3 line 671 a.g |
950 | Choice:1 Depth:1 Group:4 ("\}") |
951 | 2 #token "\}" grammar/76 line 638 a.g |
953 | Choice:2 Depth:1 Group:5 ("#errclass") |
954 | 1 in (...)* block grammar/83 line 645 a.g |
955 | 2 to error grammar/93 line 655 a.g |
956 | 3 error error/1 line 894 a.g |
957 | 4 #token "#errclass" error/2 line 895 a.g |
959 | Choice:2 Depth:1 Group:6 ("#tokclass") |
960 | 2 to tclass grammar/94 line 656 a.g |
961 | 3 tclass tclass/1 line 937 a.g |
962 | 4 #token "#tokclass" tclass/2 line 938 a.g |
964 | Choice:2 Depth:1 Group:7 ("class") |
965 | 2 to class_def grammar/95 line 657 a.g |
966 | 3 class_def class_def/1 line 669 a.g |
967 | 4 #token "class" class_def/3 line 671 a.g |
969 | Choice:2 Depth:1 Group:8 ("\}") |
970 | 2 #token "\}" grammar/96 line 658 a.g |
971 +---------------------------------------------------------------------+
973 For a linear lookahead set ambiguity (where k=1 or for k>1 but
974 when all lookahead sets [i] with i<k all have degree one) the
975 reports appear in the following order:
977 for (depth=1 ; depth <= "-aad depth" ; depth++) {
978 for (alternative=1; alternative <=2 ; alternative++) {
979 while (matches-are-found) {
986 For reporting a k-tuple ambiguity, the reports appear in the
989 for (depth=1 ; depth <= "-aad depth" ; depth++) {
990 while (matches-are-found) {
991 for (alternative=1; alternative <=2 ; alternative++) {
998 This is because matches are generated in different ways for
999 linear lookahead and k-tuples.
1001 #117. (Changed in 1.33MR10) new EXPERIMENTAL predicate hoisting code
1003 The hoisting of predicates into rules to create prediction
1004 expressions is a problem in antlr. Consider the following
1005 example (k=1 with -prc on):
1009 b : <<isUpper(LATEXT(1))>>? A ;
1012 Prior to 1.33MR10 the code generated for "start" would resemble:
1016 (!LA(1)==A || isUpper())) {
1021 This code is wrong because it makes rule "c" unreachable from
1022 "start". The essence of the problem is that antlr fails to
1023 recognize that there can be a valid alternative within "a" even
1024 when the predicate <<isUpper(LATEXT(1))>>? is false.
1026 In 1.33MR10 with -mrhoist the hoisting of the predicate into
1027 "start" is suppressed because it recognizes that "c" can
1028 cover all the cases where the predicate is false:
1036 With the antlr "-info p" switch the user will receive information
1037 about the predicate suppression in the generated file:
1039 --------------------------------------------------------------
1042 Hoisting of predicate suppressed by alternative without predicate.
1043 The alt without the predicate includes all cases where
1044 the predicate is false.
1046 WITH predicate: line 7 v1.g
1047 WITHOUT predicate: line 7 v1.g
1049 The context set for the predicate:
1053 The lookahead set for the alt WITHOUT the semantic predicate:
1059 pred << isUpper(LATEXT(1))>>?
1060 depth=k=1 rule b line 9 v1.g
1065 Chain of referenced rules:
1067 #0 in rule start (line 5 v1.g) to rule a
1068 #1 in rule a (line 7 v1.g)
1071 --------------------------------------------------------------
1073 A predicate can be suppressed by a combination of alternatives
1074 which, taken together, cover a predicate:
1078 a : b | ca | cb | cc ;
1080 b : <<isUpper(LATEXT(1))>>? ( A | B | C ) ;
1086 Consider a more complex example in which "c" covers only part of
1095 b : <<isUpper(LATEXT(1))>>?
1103 Prior to 1.33MR10 the code generated for "start" would resemble:
1106 if ( (LA(1)==A || LA(1)==X) &&
1107 (! (LA(1)==A || LA(1)==X) || isUpper()) {
1112 With 1.33MR10 and -mrhoist the predicate context is restricted to
1113 the non-covered lookahead. The code resembles:
1116 if ( (LA(1)==A || LA(1)==X) &&
1117 (! (LA(1)==X) || isUpper()) {
1122 With the antlr "-info p" switch the user will receive information
1123 about the predicate restriction in the generated file:
1125 --------------------------------------------------------------
1128 Restricting the context of a predicate because of overlap
1129 in the lookahead set between the alternative with the
1130 semantic predicate and one without
1131 Without this restriction the alternative without the predicate
1132 could not be reached when input matched the context of the
1133 predicate and the predicate was false.
1135 WITH predicate: line 11 v4.g
1136 WITHOUT predicate: line 12 v4.g
1138 The original context set for the predicate:
1142 The lookahead set for the alt WITHOUT the semantic predicate:
1146 The intersection of the two sets
1150 The original predicate:
1152 pred << isUpper(LATEXT(1))>>?
1153 depth=k=1 rule b line 15 v4.g
1158 The new (modified) form of the predicate:
1160 pred << isUpper(LATEXT(1))>>?
1161 depth=k=1 rule b line 15 v4.g
1167 --------------------------------------------------------------
1169 The bad news about -mrhoist:
1171 (a) -mrhoist does not analyze predicates with lookahead
1174 (b) -mrhoist does not look past a guarded predicate to
1175 find context which might cover other predicates.
1177 For these cases you might want to use syntactic predicates.
1178 When a semantic predicate fails during guess mode the guess
1179 fails and the next alternative is tried.
1181 Limitation (a) is illustrated by the following example:
1183 start : (stmt)* EOF ;
1188 cast : <<isTypename(LATEXT(2))>>? LP ID RP ;
1192 This is not much different from the first example, except that
1193 it requires two tokens of lookahead context to determine what
1194 to do. This predicate is NOT suppressed because the current version
1195 is unable to handle predicates with depth > 1.
1197 A predicate can be combined with other predicates during hoisting.
1198 In those cases the depth=1 predicates are still handled. Thus,
1199 in the following example the isUpper() predicate will be suppressed
1200 by line #4 when hoisted from "bizarre" into "start", but will still
1201 be present in "bizarre" in order to predict "stmt".
1203 start : (bizarre)* EOF ; // #1
1205 bizarre : stmt // #3
1213 cast : <<isTypename(LATEXT(2))>>? LP ID RP ;
1216 | <<isUpper(LATEXT(1))>>? A
1218 Limitation (b) is illustrated by the following example of a
1219 context guarded predicate:
1221 rule : (A)? <<p>>? // #1
1228 Recall that this means that when the lookahead is NOT A then
1229 the predicate "p" is ignored and it attempts to match "A|B".
1230 Ideally, the "B" at line #3 should suppress predicate "q".
1231 However, the current version does not attempt to look past
1232 the guard predicate to find context which might suppress other
1235 In some cases -mrhoist will lead to the reporting of ambiguities
1236 which were not visible before:
1242 b : <<isUpper(LATEXT(1))>>? A;
1247 In this case there is a true ambiguity in "a" between "bc" and "d"
1248 which can both match "A". Without -mrhoist the predicate in "b"
1249 is hoisted into "a" and there is no ambiguity reported. However,
1250 with -mrhoist, the predicate in "b" is suppressed by "c" (as it
1251 should be) making the ambiguity in "a" apparent.
1253 The motivations for these changes were hoisting problems reported
1254 by Reinier van den Born (reinier@vnet.ibm.com) and several others.
1256 #113. (Changed in 1.33MR10) new context guarded pred: (g)? && <<p>>? expr
1258 The existing context guarded predicate:
1260 rule : (guard)? => <<p>>? expr
1264 generates code which resembles:
1266 if (lookahead(expr) && (!guard || pred)) {
1270 This is not suitable for some applications because it allows
1271 expr() to be invoked when the predicate is false. This is
1272 intentional because it is meant to mimic automatically computed
1275 The new context guarded predicate uses the guard information
1276 differently because it has a different goal. Consider:
1278 rule : (guard)? && <<p>>? expr
1282 The new style of context guarded predicate is equivalent to:
1284 rule : <<guard==true && pred>>? expr
1288 It generates code which resembles:
1290 if (lookahead(expr) && guard && pred) {
1294 Both forms of guarded predicates severely restrict the form of
1295 the context guard: it can contain no rule references, no
1296 (...)*, no (...)+, and no {...}. It may contain token and
1297 token class references, and alternation ("|").
1299 Addition for 1.33MR11: in the token expression all tokens must
1300 be at the same height of the token tree:
1302 (A ( B | C))? && ... is ok (all height 2)
1303 (A ( B | ))? && ... is not ok (some 1, some 2)
1304 (A B C D | E F G H)? && ... is ok (all height 4)
1305 (A B C D | E )? && ... is not ok (some 4, some 1)
1307 This restriction is required in order to properly compute the lookahead
1308 set for expressions like:
1310 rule1 : (A B C)? && <<pred>>? rule2 ;
1311 rule2 : (A|X) (B|Y) (C|Z);
1313 This addition was suggested by Rienier van den Born (reinier@vnet.ibm.com)
1315 #109. (Changed in 1.33MR10) improved trace information
1317 The quality of the trace information provided by the "-gd"
1318 switch has been improved significantly. Here is an example
1319 of the output from a test program. It shows the rule name,
1320 the first token of lookahead, the call depth, and the guess
1323 exit rule gusxx {"?"} depth 2
1324 enter rule gusxx {"?"} depth 2
1325 enter rule gus1 {"o"} depth 3 guessing
1326 guess done - returning to rule gus1 {"o"} at depth 3
1327 (guess mode continues - an enclosing guess is still active)
1328 guess done - returning to rule gus1 {"Z"} at depth 3
1329 (guess mode continues - an enclosing guess is still active)
1330 exit rule gus1 {"Z"} depth 3 guessing
1331 guess done - returning to rule gusxx {"o"} at depth 2 (guess mode ends)
1332 enter rule gus1 {"o"} depth 3
1333 guess done - returning to rule gus1 {"o"} at depth 3 (guess mode ends)
1334 guess done - returning to rule gus1 {"Z"} at depth 3 (guess mode ends)
1335 exit rule gus1 {"Z"} depth 3
1336 line 1: syntax error at "Z" missing SC
1339 Rule trace reporting is controlled by the value of the integer
1340 [zz]traceOptionValue: when it is positive tracing is enabled,
1341 otherwise it is disabled. Tracing during guess mode is controlled
1342 by the value of the integer [zz]traceGuessOptionValue. When
1343 it is positive AND [zz]traceOptionValue is positive rule trace
1344 is reported in guess mode.
1346 The values of [zz]traceOptionValue and [zz]traceGuessOptionValue
1347 can be adjusted by subroutine calls listed below.
1349 Depending on the presence or absence of the antlr -gd switch
1350 the variable [zz]traceOptionValueDefault is set to 0 or 1. When
1351 the parser is initialized or [zz]traceReset() is called the
1352 value of [zz]traceOptionValueDefault is copied to [zz]traceOptionValue.
1353 The value of [zz]traceGuessOptionValue is always initialzed to 1,
1354 but, as noted earlier, nothing will be reported unless
1355 [zz]traceOptionValue is also positive.
1357 When the parser state is saved/restored the value of the trace
1358 variables are also saved/restored. If a restore causes a change in
1359 reporting behavior from on to off or vice versa this will be reported.
1361 When the -gd option is selected, the macro "#define zzTRACE_RULES"
1362 is added to appropriate output files.
1366 int traceOption(int delta)
1367 int traceGuessOption(int delta)
1369 int traceOptionValueDefault
1373 int zzTraceOption(int delta)
1374 int zzTraceGuessOption(int delta)
1376 int zzTraceOptionValueDefault
1378 The argument "delta" is added to the traceOptionValue. To
1379 turn on trace when inside a particular rule one:
1381 rule : <<traceOption(+1);>>
1385 <<traceOption(-1);>>
1386 ; /* fail clause */ <<traceOption(-1);>>
1388 One can use the same idea to turn *off* tracing within a
1389 rule by using a delta of (-1).
1391 An improvement in the rule trace was suggested by Sramji
1392 Ramanathan (ps@kumaran.com).
1394 #108. A Note on Deallocation of Variables Allocated in Guess Mode
1397 ------------------------------------------------------
1398 This mechanism only works for heap allocated variables
1399 ------------------------------------------------------
1401 The rewrite of the trace provides the machinery necessary
1402 to properly free variables or undo actions following a
1405 The macro zzUSER_GUESS_HOOK(guessSeq,zzrv) is expanded
1406 as part of the zzGUESS macro. When a guess is opened
1407 the value of zzrv is 0. When a longjmp() is executed to
1408 undo the guess, the value of zzrv will be 1.
1410 The macro zzUSER_GUESS_DONE_HOOK(guessSeq) is expanded
1411 as part of the zzGUESS_DONE macro. This is executed
1412 whether the guess succeeds or fails as part of closing
1415 The guessSeq is a sequence number which is assigned to each
1416 guess and is incremented by 1 for each guess which becomes
1417 active. It is needed by the user to associate the start of
1418 a guess with the failure and/or completion (closing) of a
1421 Guesses are nested. They must be closed in the reverse
1422 of the order that they are opened.
1424 In order to free memory used by a variable during a guess
1425 a user must write a routine which can be called to
1426 register the variable along with the current guess sequence
1427 number provided by the zzUSER_GUESS_HOOK macro. If the guess
1428 fails, all variables tagged with the corresponding guess
1429 sequence number should be released. This is ugly, but
1430 it would require a major rewrite of antlr 1.33 to use
1431 some mechanism other than setjmp()/longjmp().
1433 The order of calls for a *successful* guess would be:
1435 zzUSER_GUESS_HOOK(guessSeq,0);
1436 zzUSER_GUESS_DONE_HOOK(guessSeq);
1438 The order of calls for a *failed* guess would be:
1440 zzUSER_GUESS_HOOK(guessSeq,0);
1441 zzUSER_GUESS_HOOK(guessSeq,1);
1442 zzUSER_GUESS_DONE_HOOK(guessSeq);
1444 The default definitions of these macros are empty strings.
1446 Here is an example in C++ mode. The zzUSER_GUESS_HOOK and
1447 zzUSER_GUESS_DONE_HOOK macros and myGuessHook() routine
1448 can be used without change in both C and C++ versions.
1450 ----------------------------------------------------------------------
1455 typedef ANTLRCommonToken ANTLRToken;
1457 #include "DLGLexer.h"
1462 DLGFileInput in(stdin);
1463 DLGLexer lexer(&in,2000);
1464 ANTLRTokenBuffer pipe(&lexer,1);
1465 ANTLRCommonToken aToken;
1468 lexer.setToken(&aToken);
1483 #undef zzUSER_GUESS_HOOK
1484 #define zzUSER_GUESS_HOOK(guessSeq,zzrv) myGuessHook(guessSeq,zzrv);
1485 #undef zzUSER_GUESS_DONE_HOOK
1486 #define zzUSER_GUESS_DONE_HOOK(guessSeq) myGuessHook(guessSeq,2);
1488 void myGuessHook(int guessSeq,int zzrv) {
1490 fprintf(stderr,"User hook: starting guess #%d\n",guessSeq);
1491 } else if (zzrv == 1) {
1494 fprintf(stderr,"User hook: failed guess #%d\n",guessSeq);
1495 } else if (zzrv == 2) {
1498 fprintf(stderr,"User hook: ending guess #%d\n",guessSeq);
1505 #token "[\t \ \n]" <<skip();>>
1512 top : (which) ? <<fprintf(stderr,"%s is a which\n",s); free(s); s=NULL; >>
1513 | other <<fprintf(stderr,"%s is an other\n",s); free(s); s=NULL; >>
1514 ; <<if (s != NULL) free(s); s=NULL; >>
1522 : (label)? <<fprintf(stderr,"%s is a label\n",s);>>
1523 | (global)? <<fprintf(stderr,"%s is a global\n",s);>>
1524 | (exclamation)? <<fprintf(stderr,"%s is an exclamation\n",s);>>
1527 label : <<s=strdup(LT(1)->getText());>> A ":" ;
1529 global : <<s=strdup(LT(1)->getText());>> A "::" ;
1531 exclamation : <<s=strdup(LT(1)->getText());>> A "!" ;
1533 other : <<s=strdup(LT(1)->getText());>> "other" ;
1536 ----------------------------------------------------------------------
1538 This is a silly example, but illustrates the idea. For the input
1539 "a ::" with tracing enabled the output begins:
1541 ----------------------------------------------------------------------
1542 enter rule "start" depth 1
1543 enter rule "top" depth 2
1544 User hook: starting guess #1
1545 enter rule "which" depth 3 guessing
1546 enter rule "which2" depth 4 guessing
1547 enter rule "which3" depth 5 guessing
1548 User hook: starting guess #2
1549 enter rule "label" depth 6 guessing
1551 User hook: failed guess #2
1552 guess done - returning to rule "which3" at depth 5 (guess mode continues
1553 - an enclosing guess is still active)
1554 User hook: ending guess #2
1555 User hook: starting guess #3
1556 enter rule "global" depth 6 guessing
1557 exit rule "global" depth 6 guessing
1558 guess done - returning to rule "which3" at depth 5 (guess mode continues
1559 - an enclosing guess is still active)
1560 User hook: ending guess #3
1561 enter rule "global" depth 6 guessing
1562 exit rule "global" depth 6 guessing
1563 exit rule "which3" depth 5 guessing
1564 exit rule "which2" depth 4 guessing
1565 exit rule "which" depth 3 guessing
1566 guess done - returning to rule "top" at depth 2 (guess mode ends)
1567 User hook: ending guess #1
1568 enter rule "which" depth 3
1570 ----------------------------------------------------------------------
1574 (a) Only init-actions are executed during guess mode.
1575 (b) A rule can be invoked multiple times during guess mode.
1576 (c) If the guess succeeds the rule will be called once more
1577 without guess mode so that normal actions will be executed.
1578 This means that the init-action might need to distinguish
1579 between guess mode and non-guess mode using the variable
1582 #101. (Changed in 1.33MR10) antlr -info command line switch
1586 p - extra predicate information in generated file
1588 t - information about tnode use:
1589 at the end of each rule in generated file
1590 summary on stderr at end of program
1592 m - monitor progress
1593 prints name of each rule as it is started
1594 flushes output at start of each rule
1596 f - first/follow set information to stdout
1598 0 - no operation (added in 1.33MR11)
1600 The options may be combined and may appear in any order.
1603 antlr -info ptm -CC -gt -mrhoist on mygrammar.g
1605 #100a. (Changed in 1.33MR10) Predicate tree simplification
1607 When the same predicates can be referenced in more than one
1608 alternative of a block large predicate trees can be formed.
1610 The difference that these optimizations make is so dramatic
1611 that I have decided to use it even when -mrhoist is not selected.
1613 Consider the following grammar:
1627 c : <<AAA(LATEXT(2))>>?
1630 d : <<BBB(LATEXT(2))>>? B C
1633 e : <<CCC(LATEXT(2))>>? B C
1639 In rule "a" there is a reference to rule "c" in both alternatives.
1640 The length of the predicate AAA is k=2 and it can be followed in
1641 alternative 1 only by (A B) while in alternative 2 it can be
1642 followed only by (A C). Thus they do not have identical context.
1644 In rule "all" the alternatives which refer to rules "e" and "f" allow
1645 elimination of the duplicate reference to predicate CCC.
1647 The table below summarized the kind of simplification performed by
1648 1.33MR10. In the table, X and Y stand for single predicates
1651 (OR X (OR Y (OR Z))) => (OR X Y Z)
1652 (AND X (AND Y (AND Z))) => (AND X Y Z)
1654 (OR X (... (OR X Y) ... )) => (OR X (... Y ... ))
1655 (AND X (... (AND X Y) ... )) => (AND X (... Y ... ))
1656 (OR X (... (AND X Y) ... )) => (OR X (... ... ))
1657 (AND X (... (OR X Y) ... )) => (AND X (... ... ))
1662 In a test with a complex grammar for a real application, a predicate
1663 tree with six OR nodes and 12 leaves was reduced to "(OR X Y Z)".
1665 In 1.33MR10 there is a greater effort to release memory used
1666 by predicates once they are no longer in use.
1668 #100b. (Changed in 1.33MR10) Suppression of extra predicate tests
1670 The following optimizations require that -mrhoist be selected.
1672 It is relatively easy to optimize the code generated for predicate
1673 gates when they are of the form:
1678 where X, Y, Z, and "..." represent individual predicates (leaves) not
1681 If the predicate is an AND the contexts of the X, Y, Z, etc. are
1682 ANDed together to create a single Tree context for the group and
1683 context tests for the individual predicates are suppressed:
1685 --------------------------------------------------
1686 Note: This was incorrect. The contexts should be
1687 ORed together. This has been fixed. A more
1688 complete description is available in item #152.
1689 ---------------------------------------------------
1691 Optimization 1: (AND X Y Z ...)
1693 Suppose the context for Xtest is LA(1)==LP and the context for
1694 Ytest is LA(1)==LP && LA(2)==ID.
1696 Without the optimization the code would resemble:
1698 if (lookaheadContext &&
1699 !(LA(1)==LP && LA(1)==LP && LA(2)==ID) ||
1700 ( (! LA(1)==LP || Xtest) &&
1701 (! (LA(1)==LP || LA(2)==ID) || Xtest)
1704 With the -mrhoist optimization the code would resemble:
1706 if (lookaheadContext &&
1707 ! (LA(1)==LP && LA(2)==ID) || (Xtest && Ytest) {...
1709 Optimization 2: (OR X Y Z ...) with identical contexts
1711 Suppose the context for Xtest is LA(1)==ID and for Ytest
1712 the context is also LA(1)==ID.
1714 Without the optimization the code would resemble:
1716 if (lookaheadContext &&
1717 ! (LA(1)==ID || LA(1)==ID) ||
1718 (LA(1)==ID && Xtest) ||
1719 (LA(1)==ID && Ytest) {...
1721 With the -mrhoist optimization the code would resemble:
1723 if (lookaheadContext &&
1724 (! LA(1)==ID) || (Xtest || Ytest) {...
1726 Optimization 3: (OR X Y Z ...) with distinct contexts
1728 Suppose the context for Xtest is LA(1)==ID and for Ytest
1729 the context is LA(1)==LP.
1731 Without the optimization the code would resemble:
1733 if (lookaheadContext &&
1734 ! (LA(1)==ID || LA(1)==LP) ||
1735 (LA(1)==ID && Xtest) ||
1736 (LA(1)==LP && Ytest) {...
1738 With the -mrhoist optimization the code would resemble:
1740 if (lookaheadContext &&
1742 (LA(1)==ID && (zzpf=1) && Xtest) ||
1743 (LA(1)==LP && (zzpf=1) && Ytest) ||
1746 These may appear to be of similar complexity at first,
1747 but the non-optimized version contains two tests of each
1748 context while the optimized version contains only one
1749 such test, as well as eliminating some of the inverted
1750 logic (" !(...) || ").
1752 Optimization 4: Computation of predicate gate trees
1754 When generating code for the gates of predicate expressions
1755 antlr 1.33 vanilla uses a recursive procedure to generate
1756 "&&" and "||" expressions for testing the lookahead. As each
1757 layer of the predicate tree is exposed a new set of "&&" and
1758 "||" expressions on the lookahead are generated. In many
1759 cases the lookahead being tested has already been tested.
1761 With -mrhoist a lookahead tree is computed for the entire
1762 lookahead expression. This means that predicates with identical
1763 context or context which is a subset of another predicate's
1766 This is especially important for predicates formed by rules
1769 uppperCaseVowel : <<isUpperCase(LATEXT(1))>>? vowel;
1770 vowel: : <<isVowel(LATEXT(1))>>? LETTERS;
1772 These predicates are combined using AND since both must be
1773 satisfied for rule upperCaseVowel. They have identical
1774 context which makes this optimization very effective.
1776 The affect of Items #100a and #100b together can be dramatic. In
1777 a very large (but real world) grammar one particular predicate
1778 expression was reduced from an (unreadable) 50 predicate leaves,
1779 195 LA(1) terms, and 5500 characters to an (easily comprehensible)
1780 3 predicate leaves (all different) and a *single* LA(1) term.
1782 #98. (Changed in 1.33MR10) Option "-info p"
1784 When the user selects option "-info p" the program will generate
1785 detailed information about predicates. If the user selects
1786 "-mrhoist on" additional detail will be provided explaining
1787 the promotion and suppression of predicates. The output is part
1788 of the generated file and sandwiched between #if 0/#endif statements.
1790 Consider the following k=1 grammar:
1806 b : <<LATEXT(1)>>? X
1809 Below is an excerpt of the output for rule "start" for the three
1810 predicate options (off, on, and maintenance release style hoisting).
1812 For those who do not wish to use the "-mrhoist on" option for code
1813 generation the option can be used in a "diagnostic" mode to provide
1814 valuable information:
1816 a. where one should insert null actions to inhibit hoisting
1817 b. a chain of rule references which shows where predicates are
1820 ======================================================================
1821 Example of "-info p" with "-mrhoist on"
1822 ======================================================================
1825 Hoisting of predicate suppressed by alternative without predicate.
1826 The alt without the predicate includes all cases where the
1829 WITH predicate: line 11 v36.g
1830 WITHOUT predicate: line 12 v36.g
1832 The context set for the predicate:
1836 The lookahead set for alt WITHOUT the semantic predicate:
1842 pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g
1848 Chain of referenced rules:
1850 #0 in rule start (line 1 v36.g) to rule all
1851 #1 in rule all (line 3 v36.g) to rule a
1852 #2 in rule a (line 8 v36.g) to rule c
1853 #3 in rule c (line 11 v36.g)
1859 pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g
1866 ======================================================================
1867 Example of "-info p" with the default -prc setting ( "-prc off")
1868 ======================================================================
1872 pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g
1878 pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g
1885 ======================================================================
1886 Example of "-info p" with "-prc on" and "-mrhoist off"
1887 ======================================================================
1891 pred << LATEXT(1)>>? depth=k=1 rule c line 11 v36.g
1897 pred << LATEXT(1)>>? depth=k=1 rule b line 15 v36.g
1904 ======================================================================
1906 #60. (Changed in 1.33MR7) Major changes to exception handling
1908 There were significant problems in the handling of exceptions
1909 in 1.33 vanilla. The general problem is that it can only
1910 process one level of exception handler. For example, a named
1911 exception handler, an exception handler for an alternative, or
1912 an exception for a subrule always went to the rule's exception
1913 handler if there was no "catch" which matched the exception.
1915 In 1.33MR7 the exception handlers properly "nest". If an
1916 exception handler does not have a matching "catch" then the
1917 nextmost outer exception handler is checked for an appropriate
1918 "catch" clause, and so on until an exception handler with an
1919 appropriate "catch" is found.
1921 There are still undesirable features in the way exception
1922 handlers are implemented, but I do not have time to fix them
1925 The exception handlers for alternatives are outside the
1926 block containing the alternative. This makes it impossible
1927 to access variables declared in a block or to resume the
1928 parse by "falling through". The parse can still be easily
1929 resumed in other ways, but not in the most natural fashion.
1931 This results in an inconsistentcy between named exception
1932 handlers and exception handlers for alternatives. When
1933 an exception handler for an alternative "falls through"
1934 it goes to the nextmost outer handler - not the "normal
1937 A major difference between 1.33MR7 and 1.33 vanilla is
1938 the default action after an exception is caught:
1942 In 1.33 vanilla the signal value is set to zero ("NoSignal")
1943 and the code drops through to the code following the exception.
1944 For named exception handlers this is the "normal action".
1945 For alternative exception handlers this is the rule's handler.
1949 In 1.33MR7 the signal value is NOT automatically set to zero.
1951 There are two cases:
1953 For named exception handlers: if the signal value has been
1954 set to zero the code drops through to the "normal action".
1956 For all other cases the code branches to the nextmost outer
1957 exception handler until it reaches the handler for the rule.
1959 The following macros have been defined for convenience:
1962 --------------------
1964 set signal & return signal arg to 0 ("NoSignal")
1965 (zz)setSignal(intValue)
1966 set signal & return signal arg to some value
1968 copy the signal value to the return signal arg
1970 I'm not sure why PCCTS make a distinction between the local
1971 signal value and the return signal argument, but I'm loathe
1972 to change the code. The burden of copying the local signal
1973 value to the return signal argument can be given to the
1974 default signal handler, I suppose.
1976 #53. (Explanation for 1.33MR6) What happens after an exception is caught ?
1978 The Book is silent about what happens after an exception
1981 The following code fragment prints "Error Action" followed
1984 test : Word ex:Number <<printf("Normal Action\n");>>
1987 <<printf("Error Action\n");>>
1990 The reason for "Normal Action" is that the normal flow of the
1991 program after a user-written exception handler is to "drop through".
1992 In the case of an exception handler for a rule this results in
1993 the exection of a "return" statement. In the case of an
1994 exception handler attached to an alternative, rule, or token
1995 this is the code that would have executed had there been no
1998 The user can achieve the desired result by using a "return"
2001 test : Word ex:Number <<printf("Normal Action\n");>>
2004 <<printf("Error Action\n"); return;>>
2007 The most powerful mechanism for recovery from parse errors
2008 in pccts is syntactic predicates because they provide
2009 backtracking. Exceptions allow "return", "break",
2010 "consumeUntil(...)", "goto _handler", "goto _fail", and
2011 changing the _signal value.
2013 #41. (Added in 1.33MR6) antlr -stdout
2015 Using "antlr -stdout ..." forces the text that would
2016 normally go to the grammar.c or grammar.cpp file to
2019 #40. (Added in 1.33MR6) antlr -tab to change tab stops
2021 Using "antlr -tab number ..." changes the tab stops
2022 for the grammar.c or grammar.cpp file. The number
2023 must be between 0 and 8. Using 0 gives tab characters,
2024 values between 1 and 8 give the appropriate number of
2027 #34. (Added to 1.33MR1) Add public DLGLexerBase::set_line(int newValue)
2029 Previously there was no public function for changing the line
2030 number maintained by the lexer.
2032 #28. (Added to 1.33MR1) More control over DLG header
2034 Version 1.33MR1 adds the following directives to PCCTS
2037 #lexprefix <<source code>>
2039 Adds source code to the DLGLexer.h file
2040 after the #include "DLexerBase.h" but
2041 before the start of the class definition.
2043 #lexmember <<source code>>
2045 Adds source code to the DLGLexer.h file
2046 as part of the DLGLexer class body. It
2047 appears immediately after the start of
2048 the class and a "public: statement.