Add in the 1st version of ECP.

[mirror_edk2.git] / EdkCompatibilityPkg / Other / Maintained / Tools / Pccts / CHANGES_FROM_133.txt

=======================================================================
List of Implemented Fixes and Changes for Maintenance Releases of PCCTS


 For a summary of the most significant changes see CHANGES_SUMMARY.TXT

=======================================================================

                               DISCLAIMER

 The software and these notes are provided "as is".  They may include
 typographical or technical errors and their authors disclaims all
 liability of any kind or nature for damages due to error, fault,
 defect, or deficiency regardless of cause.  All warranties of any
 kind, either express or implied, including, but not limited to, the
 implied  warranties of merchantability and fitness for a particular
 purpose are disclaimed.


        -------------------------------------------------------
        Note:  Items #153 to #1 are now in a separate file named
                CHANGES_FROM_133_BEFORE_MR13.txt
        -------------------------------------------------------

#261. (Changed in MR19) Defer token fetch for C++ mode

    Item #216 has been revised to indicate that use of the defer fetch
    option (ZZDEFER_FETCH) requires dlg option -i.

#260. (MR22) Raise default lex buffer size from 8,000 to 32,000 bytes.

    ZZLEXBUFSIZE is the size (in bytes) of the buffer used by dlg 
    generated lexers.  The default value has been raised to 32,000 and
    the value used by antlr, dlg, and sorcerer has also been raised to
    32,000.

#259. (MR22) Default function arguments in C++ mode.

    If a rule is declared:

            rr [int i = 0] : ....

    then the declaration generated by pccts resembles:

            void rr(int i = 0);

    however, the definition must omit the default argument:

            void rr(int i) {...}

    In the past the default value was not omitted.  In MR22
    the generated code resembles:

            void rr(int i /* = 0 */ ) {...}

    Implemented by Volker H. Simonis (simonis@informatik.uni-tuebingen.de)

#258. (MR22)  Using a base class for your parser

    In item #102 (MR10) the class statement was extended to allow one
    to specify a base class other than ANTLRParser for the generated
    parser.  It turned out that this was less than useful because
    the constructor still specified ANTLRParser as the base class.

    The class statement now uses the first identifier appearing after
    the ":" as the name of the base class.  For example:

        class MyParser : public FooParser {

    Generates in MyParser.h:

            class MyParser : public FooParser {

    Generates in MyParser.cpp something that resembles:

            MyParser::MyParser(ANTLRTokenBuffer *input) :
                                         FooParser(input,1,0,0,4)
            {
                token_tbl = _token_tbl;
                traceOptionValueDefault=1;              // MR10 turn trace ON
            }

    The base class must constructor must have a signature similar to
    that of ANTLRParser.

#257. (MR21a) Removed dlg statement that -i has no effect in C++ mode.

    This was incorrect.

#256. (MR21a) Malformed syntax graph causes crash after error message.

    In the past, certain kinds of errors in the very first grammar
    element could cause the construction of a malformed graph 
    representing the grammar.  This would eventually result in a
    fatal internal error.  The code has been changed to be more
    resistant to this particular error.

#255. (MR21a) ParserBlackBox(FILE* f) 

    This constructor set openByBlackBox to the wrong value.

    Reported by Kees Bakker (kees_bakker@tasking.nl).

#254. (MR21a) Reporting syntax error at end-of-file

    When there was a syntax error at the end-of-file the syntax
    error routine would substitute "<eof>" for the programmer's
    end-of-file symbol.  This substitution is now done only when
    the programmer does not define his own end-of-file symbol
    or the symbol begins with the character "@".

    Reported by Kees Bakker (kees_bakker@tasking.nl).

#253. (MR21) Generation of block preamble (-preamble and -preamble_first)

    The antlr option -preamble causes antlr to insert the code
    BLOCK_PREAMBLE at the start of each rule and block.  It does
    not insert code before rules references, token references, or
    actions.  By properly defining the macro BLOCK_PREAMBLE the
    user can generate code which is specific to the start of blocks.

    The antlr option -preamble_first is similar, but inserts the
    code BLOCK_PREAMBLE_FIRST(PreambleFirst_123) where the symbol
    PreambleFirst_123 is equivalent to the first set defined by
    the #FirstSetSymbol described in Item #248.

    I have not investigated how these options interact with guess
    mode (syntactic predicates).

#252. (MR21) Check for null pointer in trace routine

    When some trace options are used when the parser is generated
    without the trace enabled, the current rule name may be a
    NULL pointer.  A guard was added to check for this in
    restoreState.

    Reported by Douglas E. Forester (dougf@projtech.com).

#251. (MR21) Changes to #define zzTRACE_RULES

    The macro zzTRACE_RULES was being use to pass information to
    AParser.h.  If this preprocessor symbol was not properly
    set the first time AParser.h was #included, the declaration
    of zzTRACEdata would be omitted (it is used by the -gd option).
    Subsequent #includes of AParser.h would be skipped because of 
    the #ifdef guard, so the declaration of zzTracePrevRuleName would
    never be made.  The result was that proper compilation was very 
    order dependent.

    The declaration of zzTRACEdata was made unconditional and the
    problem of removing unused declarations will be left to optimizers.
    
    Diagnosed by Douglas E. Forester (dougf@projtech.com).

#250. (MR21) Option for EXPERIMENTAL change to error sets for blocks

    The antlr option -mrblkerr turns on an experimental feature
    which is supposed to provide more accurate syntax error messages
    for k=1, ck=1 grammars.  When used with k>1 or ck>1 grammars the
    behavior should be no worse than the current behavior.

    There is no problem with the matching of elements or the computation
    of prediction expressions in pccts.  The task is only one of listing
    the most appropriate tokens in the error message.  The error sets used
    in pccts error messages are approximations of the exact error set when
    optional elements in (...)* or (...)+ are involved.  While entirely
    correct, the error messages are sometimes not 100% accurate.  

    There is also a minor philosophical issue.  For example, suppose the
    grammar expects the token to be an optional A followed by Z, and it 
    is X.  X, of course, is neither A nor Z, so an error message is appropriate.
    Is it appropriate to say "Expected Z" ?  It is correct, it is accurate,
    but it is not complete.  

    When k>1 or ck>1 the problem of providing the exactly correct
    list of tokens for the syntax error messages ends up becoming
    equivalent to evaluating the prediction expression for the
    alternatives twice. However, for k=1 ck=1 grammars the prediction
    expression can be computed easily and evaluated cheaply, so I
    decided to try implementing it to satisfy a particular application.
    This application uses the error set in an interactive command language
    to provide prompts which list the alternatives available at that
    point in the parser.  The user can then enter additional tokens to
    complete the command line.  To do this required more accurate error 
    sets then previously provided by pccts.

    In some cases the default pccts behavior may lead to more robust error
    recovery or clearer error messages then having the exact set of tokens.
    This is because (a) features like -ge allow the use of symbolic names for
    certain sets of tokens, so having extra tokens may simply obscure things
    and (b) the error set is use to resynchronize the parser, so a good
    choice is sometimes more important than having the exact set.

    Consider the following example:

            Note:  All examples code has been abbreviated
            to the absolute minimum in order to make the
            examples concise.

        star1 : (A)* Z;

    The generated code resembles:

           old                new (with -mrblkerr)
        -------------         --------------------
        for (;;) {            for (;;) {
            match(A);           match(A);
        }                     }
        match(Z);             if (! A and ! Z) then
                                FAIL(...{A,Z}...);
                              }
                              match(Z);


        With input X
            old message: Found X, expected Z
            new message: Found X, expected A, Z

    For the example:

        star2 : (A|B)* Z;

           old                      new (with -mrblkerr)
        -------------               --------------------
        for (;;) {                  for (;;) {
          if (!A and !B) break;       if (!A and !B) break;
          if (...) {                  if (...) {
            <same ...>                  <same ...>
          }                           }
          else {                      else {
            FAIL(...{A,B,Z}...)         FAIL(...{A,B}...);
          }                           }
        }                           }
        match(B);                   if (! A and ! B and !Z) then
                                        FAIL(...{A,B,Z}...);
                                    }
                                    match(B);

        With input X
            old message: Found X, expected Z
            new message: Found X, expected A, B, Z
        With input A X
            old message: Found X, expected Z
            new message: Found X, expected A, B, Z

            This includes the choice of looping back to the
            star block.

    The code for plus blocks:

        plus1 : (A)+ Z;

    The generated code resembles:

           old                  new (with -mrblkerr)
        -------------           --------------------
        do {                    do {
          match(A);               match(A);
        } while (A)             } while (A)
        match(Z);               if (! A and ! Z) then
                                  FAIL(...{A,Z}...);
                                }
                                match(Z);

        With input A X
            old message: Found X, expected Z
            new message: Found X, expected A, Z

            This includes the choice of looping back to the
            plus block.

    For the example:

        plus2 : (A|B)+ Z;

           old                    new (with -mrblkerr)
        -------------             --------------------
        do {                        do {
          if (A) {                    <same>
            match(A);                 <same>
          } else if (B) {             <same>
            match(B);                 <same>
          } else {                    <same>
            if (cnt > 1) break;       <same>
            FAIL(...{A,B,Z}...)         FAIL(...{A,B}...);
          }                           }
          cnt++;                      <same>
        }                           }

        match(Z);                   if (! A and ! B and !Z) then
                                        FAIL(...{A,B,Z}...);
                                    }
                                    match(B);

        With input X
            old message: Found X, expected A, B, Z
            new message: Found X, expected A, B
        With input A X
            old message: Found X, expected Z
            new message: Found X, expected A, B, Z

            This includes the choice of looping back to the
            star block.
    
#249. (MR21) Changes for DEC/VMS systems

    Jean-François Piéronne (jfp@altavista.net) has updated some
    VMS related command files and fixed some minor problems related
    to building pccts under the DEC/VMS operating system.  For DEC/VMS
    users the most important differences are:

        a.  Revised makefile.vms
        b.  Revised genMMS for genrating VMS style makefiles.

#248. (MR21) Generate symbol for first set of an alternative

    pccts can generate a symbol which represents the tokens which may
    appear at the start of a block:

        rr : #FirstSetSymbol(rr_FirstSet)  ( Foo | Bar ) ;

    This will generate the symbol rr_FirstSet of type SetWordType with
    elements Foo and Bar set. The bits can be tested using code similar 
    to the following:

        if (set_el(Foo, &rr_FirstSet)) { ...

    This can be combined with the C array zztokens[] or the C++ routine
    tokenName() to get the print name of the token in the first set.

    The size of the set is given by the newly added enum SET_SIZE, a 
    protected member of the generated parser's class.  The number of
    elements in the generated set will not be exactly equal to the 
    value of SET_SIZE because of synthetic tokens created by #tokclass,
    #errclass, the -ge option, and meta-tokens such as epsilon, and
    end-of-file.

    The #FirstSetSymbol must appear immediately before a block
    such as (...)+, (...)*, and {...}, and (...).  It may not appear
    immediately before a token, a rule reference, or action.  However
    a token or rule reference can be enclosed in a (...) in order to
    make the use of #pragma FirstSetSymbol legal.

            rr_bad : #FirstSetSymbol(rr_bad_FirstSet) Foo;   //  Illegal

            rr_ok :  #FirstSetSymbol(rr_ok_FirstSet) (Foo);  //  Legal
    
    Do not confuse FirstSetSymbol sets with the sets used for testing
    lookahead. The sets used for FirstSetSymbol have one element per bit,
    so the number of bytes  is approximately the largest token number
    divided by 8.  The sets used for testing lookahead store 8 lookahead 
    sets per byte, so the length of the array is approximately the largest
    token number.

    If there is demand, a similar routine for follow sets can be added.

#247. (MR21) Misleading error message on syntax error for optional elements.

    Prior to MR21, tokens which were optional did not appear in syntax
    error messages if the block which immediately followed detected a 
    syntax error.

    Consider the following grammar which accepts Number, Word, and Other:

            rr : {Number} Word;

    For this rule the code resembles:

            if (LA(1) == Number) {
                match(Number);
                consume();
            }
            match(Word);

    Prior to MR21, the error message for input "$ a" would be:

            line 1: syntax error at "$" missing Word

    With MR21 the message will be:

            line 1: syntax error at "$" expecting Word, Number.

    The generate code resembles:

            if ( (LA(1)==Number) ) {
                zzmatch(Number);
                consume();
            }
            else {
                if ( (LA(1)==Word) ) {
                    /* nothing */
                }
                else {
                    FAIL(... message for both Number and Word ...);
                }
            }
            match(Word);
        
    The code generated for optional blocks in MR21 is slightly longer
    than the previous versions, but it should give better error messages.

    The code generated for:

            { a | b | c }

    should now be *identical* to:

            ( a | b | c | )

    which was not the case prior to MR21.

    Reported by Sue Marvin (sue@siara.com).

#246. (Changed in MR21) Use of $(MAKE) for calls to make

    Calls to make from the makefiles were replaced with $(MAKE)
    because of problems when using gmake.

    Reported with fix by Sunil K.Vallamkonda (sunil@siara.com).

#245. (Changed in MR21) Changes to genmk

    The following command line options have been added to genmk:

        -cfiles ... 
            
            To add a user's C or C++ files into makefile automatically.
            The list of files must be enclosed in apostrophes.  This
            option may be specified multiple times.

        -compiler ...
    
            The name of the compiler to use for $(CCC) or $(CC).  The
            default in C++ mode is "CC".  The default in C mode is "cc".

        -pccts_path ...

            The value for $(PCCTS), the pccts directory.  The default
            is /usr/local/pccts.

    Contributed by Tomasz Babczynski (t.babczynski@ict.pwr.wroc.pl).

#244. (Changed in MR21) Rename variable "not" in antlr.g

    When antlr.g is compiled with a C++ compiler, a variable named
    "not" causes problems.  Reported by Sinan Karasu
    (sinan.karasu@boeing.com).

#243  (Changed in MR21) Replace recursion with iteration in zzfree_ast

    Another refinement to zzfree_ast in ast.c to limit recursion.

    NAKAJIMA Mutsuki (muc@isr.co.jp).


#242.  (Changed in MR21) LineInfoFormatStr

    Added an #ifndef/#endif around LineInfoFormatStr in pcctscfg.h.

#241. (Changed in MR21) Changed macro PURIFY to a no-op

                ***********************
                *** NOT IMPLEMENTED ***
                ***********************

        The PURIFY macro was changed to a no-op because it was causing 
        problems when passing C++ objects.
    
        The old definition:
    
            #define PURIFY(r,s)     memset((char *) &(r),'\\0',(s));
    
        The new definition:
    
            #define PURIFY(r,s)     /* nothing */
#endif

#240. (Changed in MR21) sorcerer/h/sorcerer.h _MATCH and _MATCHRANGE

    Added test for NULL token pointer.

    Suggested by Peter Keller (keller@ebi.ac.uk)

#239. (Changed in MR21) C++ mode AParser::traceGuessFail

    If tracing is turned on when the code has been generated
    without trace code, a failed guess generates a trace report
    even though there are no other trace reports.  This
    make the behavior consistent with other parts of the
    trace system.

    Reported by David Wigg (wiggjd@sbu.ac.uk).

#238. (Changed in MR21) Namespace version #include files

    Changed reference from CStdio to cstdio (and other
    #include file names) in the namespace version of pccts.
    Should have known better.

#237. (Changed in MR21) ParserBlackBox(FILE*)
    
    In the past, ParserBlackBox would close the FILE in the dtor
    even though it was not opened by ParserBlackBox.  The problem
    is that there were two constructors, one which accepted a file   
    name and did an fopen, the other which accepted a FILE and did
    not do an fopen.  There is now an extra member variable which
    remembers whether ParserBlackBox did the open or not.

    Suggested by Mike Percy (mpercy@scires.com).

#236. (Changed in MR21) tmake now reports down pointer problem

    When ASTBase::tmake attempts to update the down pointer of 
    an AST it checks to see if the down pointer is NULL.  If it
    is not NULL it does not do the update and returns NULL.
    An attempt to update the down pointer is almost always a
    result of a user error.  This can lead to difficult to find
    problems during tree construction.

    With this change, the routine calls a virtual function
    reportOverwriteOfDownPointer() which calls panic to
    report the problem.  Users who want the old behavior can
    redefined the virtual function in their AST class.

    Suggested by Sinan Karasu (sinan.karasu@boeing.com)

#235. (Changed in MR21) Made ANTLRParser::resynch() virtual

    Suggested by Jerry Evans (jerry@swsl.co.uk).

#234. (Changed in MR21) Implicit int for function return value

    ATokenBuffer:bufferSize() did not specify a type for the
    return value.

    Reported by Hai Vo-Ba (hai@fc.hp.com).

#233. (Changed in MR20) Converted to MSVC 6.0

    Due to external circumstances I have had to convert to MSVC 6.0
    The MSVC 5.0 project files (.dsw and .dsp) have been retained as
    xxx50.dsp and xxx50.dsw.  The MSVC 6.0 files are named xxx60.dsp
    and xxx60.dsw (where xxx is the related to the directory/project).

#232. (Changed in MR20) Make setwd bit vectors protected in parser.h

    The access for the setwd array in the parser header was not
    specified.  As a result, it would depend on the code which 
    preceded it.  In MR20 it will always have access "protected".

    Reported by Piotr Eljasiak (eljasiak@zt.gdansk.tpsa.pl).

#231. (Changed in MR20) Error in token buffer debug code.

    When token buffer debugging is selected via the pre-processor
    symbol DEBUG_TOKENBUFFER there is an erroneous check in
    AParser.cpp:

        #ifdef DEBUG_TOKENBUFFER
            if (i >= inputTokens->bufferSize() ||
                inputTokens->minTokens() < LLk )     /* MR20 Was "<=" */
        ...
        #endif

    Reported by David Wigg (wiggjd@sbu.ac.uk).

#230. (Changed in MR20) Fixed problem with #define for -gd option

    There was an error in setting zzTRACE_RULES for the -gd (trace) option.

    Reported by Gary Funck (gary@intrepid.com).

#229. (Changed in MR20) Additional "const" for literals

    "const" was added to the token name literal table.
    "const" was added to some panic() and similar routine

#228. (Changed in MR20) dlg crashes on "()"

    The following token defintion will cause DLG to crash.

        #token "()"

    When there is a syntax error in a regular expression
    many of the dlg routines return a structure which has
    null pointers.  When this is accessed by callers it
    generates the crash.

    I have attempted to fix the more common cases.

    Reported by  Mengue Olivier (dolmen@bigfoot.com).

#227. (Changed in MR20) Array overwrite

    Steveh Hand (sassth@unx.sas.com) reported a problem which
    was traced to a temporary array which was not properly
    resized for deeply nested blocks.  This has been fixed.

#226. (Changed in MR20) -pedantic conformance
   
    G. Hobbelt (i_a@mbh.org) and THM made many, many minor 
    changes to create prototypes for all the functions and
    bring antlr, dlg, and sorcerer into conformance with
    the gcc -pedantic option.

    This may require uses to add pccts/h/pcctscfg.h to some
    files or makefiles in order to have __USE_PROTOS defined.

#225  (Changed in MR20) AST stack adjustment in C mode

    The fix in #214 for AST stack adjustment in C mode missed 
    some cases.

    Reported with fix by Ger Hobbelt (i_a@mbh.org).

#224  (Changed in MR20) LL(1) and LL(2) with #pragma approx

    This may take a record for the oldest, most trival, lexical
    error in pccts.  The regular expressions for LL(1) and LL(2)
    lacked an escape for the left and right parenthesis.

    Reported by Ger Hobbelt (i_a@mbh.org).

#223  (Changed in MR20) Addition of IBM_VISUAL_AGE directory

    Build files for antlr, dlg, and sorcerer under IBM Visual Age 
    have been contributed by Anton Sergeev (ags@mlc.ru).  They have
    been placed in the pccts/IBM_VISUAL_AGE directory.

#222  (Changed in MR20) Replace __STDC__ with __USE_PROTOS

    Most occurrences of __STDC__ replaced with __USE_PROTOS due to
    complaints from several users.

#221  (Changed in MR20) Added #include for DLexerBase.h to PBlackBox.

    Added #include for DLexerBase.h to PBlackBox.

#220  (Changed in MR19) strcat arguments reversed in #pred parse

    The arguments to strcat are reversed when creating a print
    name for a hash table entry for use with #pred feature.

    Problem diagnosed and fix reported by Scott Harrington 
    (seh4@ix.netcom.com).

#219. (Changed in MR19) C Mode routine zzfree_ast

    Changes to reduce use of recursion for AST trees with only right
    links or only left links in the C mode routine zzfree_ast.

    Implemented by SAKAI Kiyotaka (ksakai@isr.co.jp).

#218. (Changed in MR19) Changes to support unsigned char in C mode

    Changes to antlr.h and err.h to fix omissions in use of zzchar_t

    Implemented by SAKAI Kiyotaka (ksakai@isr.co.jp).

#217. (Changed in MR19) Error message when dlg -i and -CC options selected
    
    *** This change was rescinded by item #257 ***

    The parsers generated by pccts in C++ mode are not able to support the
    interactive lexer option (except, perhaps, when using the deferred fetch
    parser option.(Item #216).

    DLG now warns when both -i and -CC are selected.

    This warning was suggested by David Venditti (07751870267-0001@t-online.de).

#216. (Changed in MR19) Defer token fetch for C++ mode

    Implemented by Volker H. Simonis (simonis@informatik.uni-tuebingen.de)

    Normally, pccts keeps the lookahead token buffer completely filled.
    This requires max(k,ck) tokens of lookahead.  For some applications
    this can cause deadlock problems.  For example, there may be cases
    when the parser can't tell when the input has been completely consumed
    until the parse is complete, but the parse can't be completed because 
    the input routines are waiting for additional tokens to fill the
    lookahead buffer.
    
    When the ANTLRParser class is built with the pre-processor option 
    ZZDEFER_FETCH defined, the fetch of new tokens by consume() is deferred
    until LA(i) or LT(i) is called. 

    To test whether this option has been built into the ANTLRParser class
    use "isDeferFetchEnabled()".

    Using the -gd trace option with the default tracein() and traceout()
    routines will defeat the effort to defer the fetch because the
    trace routines print out information about the lookahead token at
    the start of the rule.
    
    Because the tracein and traceout routines are virtual it is 
    easy to redefine them in your parser:

        class MyParser {
        <<
            virtual void tracein(ANTLRChar * ruleName)
                { fprintf(stderr,"Entering: %s\n", ruleName); }
            virtual void traceout(ANTLRChar * ruleName)
                { fprintf(stderr,"Leaving: %s\n", ruleName); }
        >>
 
    The originals for those routines are pccts/h/AParser.cpp
 
    This requires use of the dlg option -i (interactive lexer).

    This is experimental.  The interaction with guess mode (syntactic
    predicates)is not known.

#215. (Changed in MR19) Addition of reset() to DLGLexerBase

    There was no obvious way to reset the lexer for reuse.  The
    reset() method now does this.

    Suggested by David Venditti (07751870267-0001@t-online.de).

#214. (Changed in MR19)  C mode: Adjust AST stack pointer at exit

    In C mode the AST stack pointer needs to be reset if there will
    be multiple calls to the ANTLRx macros.

    Reported with fix by Paul D. Smith (psmith@baynetworks.com).

#213. (Changed in MR18)  Fatal error with -mrhoistk (k>1 hoisting)

    When rearranging code I forgot to un-comment a critical line of
    code that handles hoisting of predicates with k>1 lookahead.  This
    is now fixed.

    Reported by Reinier van den Born (reinier@vnet.ibm.com).

#212. (Changed in MR17)  Mac related changes by Kenji Tanaka

    Kenji Tanaka (kentar@osa.att.ne.jp) has made a number of changes for
    Macintosh users.

    a.  The following Macintosh MPW files aid in installing pccts on Mac:

            pccts/MPW_Read_Me

            pccts/install68K.mpw
            pccts/installPPC.mpw

            pccts/antlr/antlr.r
            pccts/antlr/antlr68K.make
            pccts/antlr/antlrPPC.make

            pccts/dlg/dlg.r
            pccts/dlg/dlg68K.make
            pccts/dlg/dlgPPC.make

            pccts/sorcerer/sor.r
            pccts/sorcerer/sor68K.make
            pccts/sorcerer/sorPPC.make
    
       They completely replace the previous Mac installation files.
            
    b. The most significant is a change in the MAC_FILE_CREATOR symbol
       in pcctscfg.h:

        old: #define MAC_FILE_CREATOR 'MMCC'   /* Metrowerks C/C++ Text files */
        new: #define MAC_FILE_CREATOR 'CWIE'   /* Metrowerks C/C++ Text files */

    c.  Added calls to special_fopen_actions() where necessary.

#211. (Changed in MR16a)  C++ style comment in dlg

    This has been fixed.

#210. (Changed in MR16a)  Sor accepts \r\n, \r, or \n for end-of-line

    A user requested that Sorcerer be changed to accept other forms
    of end-of-line.

#209. (Changed in MR16) Name of files changed.

        Old:  CHANGES_FROM_1.33
        New:  CHANGES_FROM_133.txt

        Old:  KNOWN_PROBLEMS
        New:  KNOWN_PROBLEMS.txt

#208. (Changed in MR16) Change in use of pccts #include files

    There were problems with MS DevStudio when mixing Sorcerer and
    PCCTS in the same source file.  The problem is caused by the
    redefinition of setjmp in the MS header file setjmp.h.  In
    setjmp.h the pre-processor symbol setjmp was redefined to be
    _setjmp.  A later effort to execute #include <setjmp.h> resulted 
    in an effort to #include <_setjmp.h>.  I'm not sure whether this
    is a bug or a feature.  In any case, I decided to fix it by
    avoiding the use of pre-processor symbols in #include statements
    altogether.  This has the added benefit of making pre-compiled
    headers work again.

    I've replaced statements:

        old: #include PCCTS_SETJMP_H
        new: #include "pccts_setjmp.h"

    Where pccts_setjmp.h contains:

            #ifndef __PCCTS_SETJMP_H__
            #define __PCCTS_SETJMP_H__
    
            #ifdef PCCTS_USE_NAMESPACE_STD
            #include <Csetjmp>
            #else
            #include <setjmp.h>
            #endif

            #endif
        
    A similar change has been made for other standard header files
    required by pccts and sorcerer: stdlib.h, stdarg.h, stdio.h, etc.

    Reported by Jeff Vincent (JVincent@novell.com) and Dale Davis
    (DalDavis@spectrace.com).

#207. (Changed in MR16) dlg reports an invalid range for: [\0x00-\0xff]

    dlg will report that this is an invalid range.

    Diagnosed by Piotr Eljasiak (eljasiak@no-spam.zt.gdansk.tpsa.pl):

        I think this problem is not specific to unsigned chars
        because dlg reports no error for the range [\0x00-\0xfe].

        I've found that information on range is kept in field
        letter (unsigned char) of Attrib struct. Unfortunately
        the letter value internally is for some reasons increased
        by 1, so \0xff is represented here as 0.

        That's why dlg complains about the range [\0x00-\0xff] in
        dlg_p.g:

        if ($$.letter > $2.letter) {
          error("invalid range  ", zzline);
        } 

    The fix is:

        if ($$.letter > $2.letter && 255 != $$2.letter) {
          error("invalid range  ", zzline);
        } 

#206. (Changed in MR16) Free zzFAILtext in ANTLRParser destructor

    The ANTLRParser destructor now frees zzFAILtext.

    Problem and fix reported by Manfred Kogler (km@cast.uni-linz.ac.at).

#205. (Changed in MR16) DLGStringReset argument now const

    Changed: void DLGStringReset(DLGChar *s) {...}
    To:      void DLGStringReset(const DLGChar *s) {...}

    Suggested by Dale Davis (daldavis@spectrace.com)

#204. (Changed in MR15a) Change __WATCOM__ to __WATCOMC__ in pcctscfg.h
    
    Reported by Oleg Dashevskii (olegdash@my-dejanews.com).

#203. (Changed in MR15) Addition of sorcerer to distribution kit

    I have finally caved in to popular demand.  The pccts 1.33mr15
    kit will include sorcerer.  The separate sorcerer kit will be
    discontinued.

#202. (Changed) in MR15) Organization of MS Dev Studio Projects in Kit

    Previously there was one workspace that contained projects for
    all three parts of pccts: antlr, dlg, and sorcerer.  Now each
    part (and directory) has its own workspace/project and there
    is an additional workspace/project to build a library from the
    .cpp files in the pccts/h directory.

    The library build will create pccts_debug.lib or pccts_release.lib
    according to the configuration selected.  

    If you don't want to build pccts 1.33MR15 you can download a
    ready-to-run kit for win32 from http://www.polhode.com/win32.zip.
    The ready-to-run for win32 includes executables, a pre-built static
    library for the .cpp files in the pccts/h directory, and a  sample
    application

    You will need to define the environment variable PCCTS to point to
    the root of the pccts directory hierarchy.

#201. (Changed in MR15) Several fixes by K.J. Cummings (cummings@peritus.com)

      Generation of SETJMP rather than SETJMP_H in gen.c.

      (Sor B19) Declaration of ref_vars_inits for ref_var_inits in
      pccts/sorcerer/sorcerer.h.

#200. (Changed in MR15) Remove operator=() in AToken.h

      User reported that WatCom couldn't handle use of
      explicit operator =().  Replace with equivalent
      using cast operator.

#199. (Changed in MR15) Don't allow use of empty #tokclass

      Change antlr.g to disallow empty #tokclass sets.

      Reported by Manfred Kogler (km@cast.uni-linz.ac.at).

#198. Revised ANSI C grammar due to efforts by Manuel Kessler

      Manuel Kessler (mlkessler@cip.physik.uni-wuerzburg.de)

          Allow trailing ... in function parameter lists.
          Add bit fields.
          Allow old-style function declarations.
          Support cv-qualified pointers.
          Better checking of combinations of type specifiers.
          Release of memory for local symbols on scope exit.
          Allow input file name on command line as well as by redirection.

              and other miscellaneous tweaks.

      This is not part of the pccts distribution kit. It must be
      downloaded separately from:

            http://www.polhode.com/ansi_mr15.zip

#197. (Changed in MR14) Resetting the lookahead buffer of the parser

      Explanation and fix by Sinan Karasu (sinan.karasu@boeing.com)

      Consider the code used to prime the lookahead buffer LA(i)
      of the parser when init() is called:

        void
        ANTLRParser::
        prime_lookahead()
        {
            int i;
            for(i=1;i<=LLk; i++) consume();
            dirty=0;
            //lap = 0;      // MR14 - Sinan Karasu (sinan.karusu@boeing.com)
            //labase = 0;   // MR14
            labase=lap;     // MR14
        }

      When the parser is instantiated, lap=0,labase=0 is set.

      The "for" loop runs LLk times. In consume(), lap = lap +1 (mod LLk) is
      computed.  Therefore, lap(before the loop) == lap (after the loop).

      Now the only problem comes in when one does an init() of the parser
      after an Eof has been seen. At that time, lap could be non zero.
      Assume it was lap==1. Now we do a prime_lookahead(). If LLk is 2,
      then

        consume()
        {
            NLA = inputTokens->getToken()->getType();
            dirty--;
            lap = (lap+1)&(LLk-1);
        }

      or expanding NLA,

        token_type[lap&(LLk-1)]) = inputTokens->getToken()->getType();
        dirty--;
        lap = (lap+1)&(LLk-1);

      so now we prime locations 1 and 2.  In prime_lookahead it used to set
      lap=0 and labase=0.  Now, the next token will be read from location 0,
      NOT 1 as it should have been.

      This was never caught before, because if a parser is just instantiated,
      then lap and labase are 0, the offending assignment lines are
      basically no-ops, since the for loop wraps around back to 0.

#196. (Changed in MR14) Problems with "(alpha)? beta" guess

    Consider the following syntactic predicate in a grammar
    with 2 tokens of lookahead (k=2 or ck=2):

        rule  : ( alpha )? beta ;
        alpha : S t ;
        t     : T U
              | T
              ;
        beta  : S t Z ;

    When antlr computes the prediction expression with one token
    of lookahead for alts 1 and 2 of rule t it finds an ambiguity.

    Because the grammar has a lookahead of 2 it tries to compute
    two tokens of lookahead for alts 1 and 2 of t.  Alt 1 clearly
    has a lookahead of (T U).  Alt 2 is one token long so antlr
    tries to compute the follow set of alt 2, which means finding
    the things which can follow rule t in the context of (alpha)?.
    This cannot be computed, because alpha is only part of a rule,
    and antlr can't tell what part of beta is matched by alpha and
    what part remains to be matched.  Thus it impossible for antlr
    to  properly determine the follow set of rule t.

    Prior to 1.33MR14, the follow of (alpha)? was computed as
    FIRST(beta) as a result of the internal representation of
    guess blocks.

    With MR14 the follow set will be the empty set for that context.

    Normally, one expects a rule appearing in a guess block to also
    appear elsewhere.  When the follow context for this other use
    is "ored" with the empty set, the context from the other use
    results, and a reasonable follow context results.  However if
    there is *no* other use of the rule, or it is used in a different
    manner then the follow context will be inaccurate - it was
    inaccurate even before MR14, but it will be inaccurate in a
    different way.

    For the example given earlier, a reasonable way to rewrite the
    grammar:

        rule  : ( alpha )? beta
        alpha : S t ;
        t     : T U
              | T
              ;
        beta  : alpha Z ;

    If there are no other uses of the rule appearing in the guess
    block it will generate a test for EOF - a workaround for
    representing a null set in the lookahead tests.

    If you encounter such a problem you can use the -alpha option
    to get additional information:

    line 2: error: not possible to compute follow set for alpha
              in an "(alpha)? beta" block.

    With the antlr -alpha command line option the following information
    is inserted into the generated file:

    #if 0

      Trace of references leading to attempt to compute the follow set of
      alpha in an "(alpha)? beta" block. It is not possible for antlr to
      compute this follow set because it is not known what part of beta has
      already been matched by alpha and what part remains to be matched.

      Rules which make use of the incorrect follow set will also be incorrect

         1 #token T              alpha/2   line 7     brief.g
         2 end alpha             alpha/3   line 8     brief.g
         2 end (...)? block at   start/1   line 2     brief.g

    #endif

    At the moment, with the -alpha option selected the program marks
    any rules which appear in the trace back chain (above) as rules with
    possible problems computing follow set.

    Reported by Greg Knapen (gregory.knapen@bell.ca).

#195. (Changed in MR14) #line directive not at column 1

      Under certain circunstances a predicate test could generate
      a #line directive which was not at column 1.

      Reported with fix by David Kågedal  (davidk@lysator.liu.se)
      (http://www.lysator.liu.se/~davidk/).

#194. (Changed in MR14) (C Mode only) Demand lookahead with #tokclass

      In C mode with the demand lookahead option there is a bug in the
      code which handles matches for #tokclass (zzsetmatch and
      zzsetmatch_wsig).

      The bug causes the lookahead pointer to get out of synchronization
      with the current token pointer.

      The problem was reported with a fix by Ger Hobbelt (hobbelt@axa.nl).

#193. (Changed in MR14) Use of PCCTS_USE_NAMESPACE_STD

      The pcctscfg.h now contains the following definitions:

        #ifdef PCCTS_USE_NAMESPACE_STD
        #define PCCTS_STDIO_H     <Cstdio>
        #define PCCTS_STDLIB_H    <Cstdlib>
        #define PCCTS_STDARG_H    <Cstdarg>
        #define PCCTS_SETJMP_H    <Csetjmp>
        #define PCCTS_STRING_H    <Cstring>
        #define PCCTS_ASSERT_H    <Cassert>
        #define PCCTS_ISTREAM_H   <istream>
        #define PCCTS_IOSTREAM_H  <iostream>
        #define PCCTS_NAMESPACE_STD     namespace std {}; using namespace std;
        #else
        #define PCCTS_STDIO_H     <stdio.h>
        #define PCCTS_STDLIB_H    <stdlib.h>
        #define PCCTS_STDARG_H    <stdarg.h>
        #define PCCTS_SETJMP_H    <setjmp.h>
        #define PCCTS_STRING_H    <string.h>
        #define PCCTS_ASSERT_H    <assert.h>
        #define PCCTS_ISTREAM_H   <istream.h>
        #define PCCTS_IOSTREAM_H  <iostream.h>
        #define PCCTS_NAMESPACE_STD
        #endif

      The runtime support in pccts/h uses these pre-processor symbols
      consistently.

      Also, antlr and dlg have been changed to generate code which uses
      these pre-processor symbols rather than having the names of the
      #include files hard-coded in the generated code.

      This required the addition of "#include pcctscfg.h" to a number of
      files in pccts/h.

      It appears that this sometimes causes problems for MSVC 5 in
      combination with the "automatic" option for pre-compiled headers.
      In such cases disable the "automatic" pre-compiled headers option.

      Suggested by Hubert Holin (Hubert.Holin@Bigfoot.com).

#192. (Changed in MR14) Change setText() to accept "const ANTLRChar *"

      Changed ANTLRToken::setText(ANTLRChar *) to setText(const ANTLRChar *).
      This allows literal strings to be used to initialize tokens.  Since
      the usual token implementation (ANTLRCommonToken)  makes a copy of the
      input string, this was an unnecessary limitation.

      Suggested by Bob McWhirter (bob@netwrench.com).

#191. (Changed in MR14) HP/UX aCC compiler compatibility problem

      Needed to explicitly declare zzINF_DEF_TOKEN_BUFFER_SIZE and
      zzINF_BUFFER_TOKEN_CHUNK_SIZE as ints in pccts/h/AParser.cpp.

      Reported by David Cook (dcook@bmc.com).

#190. (Changed in MR14) IBM OS/2 CSet compiler compatibility problem

      Name conflict with "_cs" in pccts/h/ATokenBuffer.cpp

      Reported by David Cook (dcook@bmc.com).

#189. (Changed in MR14) -gxt switch in C mode

      The -gxt switch in C mode didn't work because of incorrect
      initialization.

      Reported by Sinan Karasu (sinan@boeing.com).

#188. (Changed in MR14) Added pccts/h/DLG_stream_input.h

      This is a DLG stream class based on C++ istreams.

      Contributed by Hubert Holin (Hubert.Holin@Bigfoot.com).

#187. (Changed in MR14) Rename config.h to pcctscfg.h

      The PCCTS configuration file has been renamed from config.h to
      pcctscfg.h.  The problem with the original name is that it led
      to name collisions when pccts parsers were combined with other
      software.

      All of the runtime support routines in pccts/h/* have been
      changed to use the new name.  Existing software can continue
      to use pccts/h/config.h. The contents of pccts/h/config.h is
      now just "#include "pcctscfg.h".

      I don't have a record of the user who suggested this.

#186. (Changed in MR14) Pre-processor symbol DllExportPCCTS class modifier

      Classes in the C++ runtime support routines are now declared:

        class DllExportPCCTS className ....

      By default, the pre-processor symbol is defined as the empty
      string.  This if for use by MSVC++ users to create DLL classes.

      Suggested by Manfred Kogler (km@cast.uni-linz.ac.at).

#185. (Changed in MR14) Option to not use PCCTS_AST base class for ASTBase

      Normally, the ASTBase class is derived from PCCTS_AST which contains
      functions useful to Sorcerer.  If these are not necessary then the
      user can define the pre-processor symbol "PCCTS_NOT_USING_SOR" which
      will cause the ASTBase class to replace references to PCCTS_AST with
      references to ASTBase where necessary.

      The class ASTDoublyLinkedBase will contain a pure virtual function
      shallowCopy() that was formerly defined in class PCCTS_AST.

      Suggested by Bob McWhirter (bob@netwrench.com).

#184. (Changed in MR14) Grammars with no tokens generate invalid tokens.h

      Reported by Hubert Holin (Hubert.Holin@bigfoot.com).

#183. (Changed in MR14) -f to specify file with names of grammar files

      In DEC/VMS it is difficult to specify very long command lines.
      The -f option allows one to place the names of the grammar files
      in a data file in order to bypass limitations of the DEC/VMS
      command language interpreter.

      Addition supplied by Bernard Giroud (b_giroud@decus.ch).

#182. (Changed in MR14) Output directory option for DEC/VMS

      Fix some problems with the -o option under DEC/VMS.

      Fix supplied by Bernard Giroud (b_giroud@decus.ch).

#181. (Changed in MR14) Allow chars > 127 in DLGStringInput::nextChar()

      Changed DLGStringInput to cast the character using (unsigned char)
      so that languages with character codes greater than 127 work
      without changes.

      Suggested by Manfred Kogler (km@cast.uni-linz.ac.at).

#180. (Added in MR14) ANTLRParser::getEofToken()

      Added "ANTLRToken ANTLRParser::getEofToken() const" to match the
      setEofToken routine.

      Requested by Manfred Kogler (km@cast.uni-linz.ac.at).

#179. (Fixed in MR14) Memory leak for BufFileInput subclass of DLGInputStream

      The BufFileInput class described in Item #142 neglected to release
      the allocated buffer when an instance was destroyed.

      Reported by Manfred Kogler (km@cast.uni-linz.ac.at).

#178. (Fixed in MR14) Bug in "(alpha)? beta" guess blocks first sets

      In 1.33 vanilla, and all maintenance releases prior to MR14
      there is a bug in the handling of guess blocks which use the
      "long" form:

                  (alpha)? beta

      inside a (...)*, (...)+, or {...} block.

      This problem does *not* apply to the case where beta is omitted
      or when the syntactic predicate is on the leading edge of an
      alternative.

      The problem is that both alpha and beta are stored in the
      syntax diagram, and that some analysis routines would fail
      to skip the alpha portion when it was not on the leading edge.
      Consider the following grammar with -ck 2:

                r : ( (A)? B )* C D

                  | A B      /* forces -ck 2 computation for old antlr    */
                             /*              reports ambig for alts 1 & 2 */

                  | B C      /* forces -ck 2 computation for new antlr    */
                             /*              reports ambig for alts 1 & 3 */
                  ;

      The prediction expression for the first alternative should be
      LA(1)={B C} LA(2)={B C D}, but previous versions of antlr
      would compute the prediction expression as LA(1)={A C} LA(2)={B D}

      Reported by Arpad Beszedes (beszedes@inf.u-szeged.hu) who provided
      a very clear example of the problem and identified the probable cause.

#177. (Changed in MR14) #tokdefs and #token with regular expression

      In MR13 the change described by Item #162 caused an existing
      feature of antlr to fail.  Prior to the change it was possible
      to give regular expression definitions and actions to tokens
      which were defined via the #tokdefs directive.

      This now works again.

      Reported by Manfred Kogler (km@cast.uni-linz.ac.at).

#176. (Changed in MR14) Support for #line in antlr source code

      Note: this was implemented by Arpad Beszedes (beszedes@inf.u-szeged.hu).

      In 1.33MR14 it is possible for a pre-processor to generate #line
      directives in the antlr source and have those line numbers and file
      names used in antlr error messages and in the #line directives
      generated by antlr.

      The #line directive may appear in the following forms:

            #line ll "sss" xx xx ...

      where ll represents a line number, "sss" represents the name of a file
      enclosed in quotation marks, and xxx are arbitrary integers.

      The following form (without "line") is not supported at the moment:

            # ll "sss" xx xx ...

      The result:

        zzline

            is replaced with ll from the # or #line directive

        FileStr[CurFile]

            is updated with the contents of the string (if any)
            following the line number

      Note
      ----
      The file-name string following the line number can be a complete
      name with a directory-path. Antlr generates the output files from
      the input file name (by replacing the extension from the file-name
      with .c or .cpp).

      If the input file (or the file-name from the line-info) contains
      a path:

        "../grammar.g"

      the generated source code will be placed in "../grammar.cpp" (i.e.
      in the parent directory).  This is inconvenient in some cases
      (even the -o switch can not be used) so the path information is
      removed from the #line directive.  Thus, if the line-info was

        #line 2 "../grammar.g"

      then the current file-name will become "grammar.g"

      In this way, the generated source code according to the grammar file
      will always be in the current directory, except when the -o switch
      is used.

#175. (Changed in MR14) Bug when guess block appears at start of (...)*

      In 1.33 vanilla and all maintenance releases prior to 1.33MR14
      there is a bug when a guess block appears at the start of a (...)+.
      Consider the following k=1 (ck=1) grammar:

            rule :
                  ( (STAR)? ZIP )* ID ;

      Prior to 1.33MR14, the generated code resembled:

        ...
        zzGUESS_BLOCK
        while ( 1 ) {
            if ( ! LA(1)==STAR) break;
            zzGUESS
            if ( !zzrv ) {
                zzmatch(STAR);
                zzCONSUME;
                zzGUESS_DONE
                zzmatch(ZIP);
                zzCONSUME;
            ...

      Note that the routine uses STAR for the prediction expression
      rather than ZIP.  With 1.33MR14 the generated code resembles:

        ...
        while ( 1 ) {
            if ( ! LA(1)==ZIP) break;
        ...

      This problem existed only with (...)* blocks and was caused
      by the slightly more complicate graph which represents (...)*
      blocks.  This caused the analysis routine to compute the first
      set for the alpha part of the "(alpha)? beta" rather than the
      beta part.

      Both (...)+ and {...} blocks handled the guess block correctly.

      Reported by Arpad Beszedes (beszedes@inf.u-szeged.hu) who provided
      a very clear example of the problem and identified the probable cause.

#174. (Changed in MR14) Bug when action precedes syntactic predicate

      In 1.33 vanilla, and all maintenance releases prior to MR14,
      there was a bug when a syntactic predicate was immediately
      preceded by an action.  Consider the following -ck 2 grammar:

            rule :
                   <<int i;>>
                   (alpha)? beta C
                 | A B
                 ;

            alpha : A ;
            beta  : A B;

      Prior to MR14, the code generated for the first alternative
      resembled:

        ...
        zzGUESS
        if ( !zzrv && LA(1)==A && LA(2)==A) {
            alpha();
            zzGUESS_DONE
            beta();
            zzmatch(C);
            zzCONSUME;
        } else {
        ...

      The prediction expression (i.e. LA(1)==A && LA(2)==A) is clearly
      wrong because LA(2) should be matched to B (first[2] of beta is {B}).

      With 1.33MR14 the prediction expression is:

        ...
        if ( !zzrv && LA(1)==A && LA(2)==B) {
            alpha();
            zzGUESS_DONE
            beta();
            zzmatch(C);
            zzCONSUME;
        } else {
        ...

      This will only affect users in which alpha is shorter than
      than max(k,ck) and there is an action immediately preceding
      the syntactic predicate.

      This problem was reported by reported by Arpad Beszedes
      (beszedes@inf.u-szeged.hu) who provided a very clear example
      of the problem and identified the presence of the init-action
      as the likely culprit.

#173. (Changed in MR13a) -glms for Microsoft style filenames with -gl

      With the -gl option antlr generates #line directives using the
      exact name of the input files specified on the command line.
      An oddity of the Microsoft C and C++ compilers is that they
      don't accept file names in #line directives containing "\"
      even though these are names from the native file system.

      With -glms option, the "\" in file names appearing in #line
      directives is replaced with a "/" in order to conform to
      Microsoft compiler requirements.

      Reported by Erwin Achermann (erwin.achermann@switzerland.org).

#172. (Changed in MR13) \r\n in antlr source counted as one line

      Some MS software uses \r\n to indicate a new line.  Antlr
      now recognizes this in counting lines.

      Reported by Edward L. Hepler (elh@ece.vill.edu).

#171. (Changed in MR13) #tokclass L..U now allowed

      The following is now allowed:

            #tokclass ABC { A..B C }

      Reported by Dave Watola (dwatola@amtsun.jpl.nasa.gov)

#170. (Changed in MR13) Suppression for predicates with lookahead depth >1

      In MR12 the capability for suppression of predicates with lookahead
      depth=1 was introduced.  With MR13 this had been extended to
      predicates with lookahead depth > 1 and released for use by users
      on an experimental basis.

      Consider the following grammar with -ck 2 and the predicate in rule
      "a" with depth 2:

            r1  : (ab)* "@"
                ;

            ab  : a
                | b
                ;

            a   : (A B)? => <<p(LATEXT(2))>>? A B C
                ;

            b   : A B C
                ;

      Normally, the predicate would be hoisted into rule r1 in order to
      determine whether to call rule "ab".  However it should *not* be
      hoisted because, even if p is false, there is a valid alternative
      in rule b.  With "-mrhoistk on" the predicate will be suppressed.

      If "-info p" command line option is present the following information
      will appear in the generated code:

                while ( (LA(1)==A)
        #if 0

        Part (or all) of predicate with depth > 1 suppressed by alternative
            without predicate

        pred  <<  p(LATEXT(2))>>?
                  depth=k=2  ("=>" guard)  rule a  line 8  t1.g
          tree context:
            (root = A
               B
            )

        The token sequence which is suppressed: ( A B )
        The sequence of references which generate that sequence of tokens:

           1 to ab          r1/1       line 1     t1.g
           2 ab             ab/1       line 4     t1.g
           3 to b           ab/2       line 5     t1.g
           4 b              b/1        line 11    t1.g
           5 #token A       b/1        line 11    t1.g
           6 #token B       b/1        line 11    t1.g

        #endif

      A slightly more complicated example:

            r1  : (ab)* "@"
                ;

            ab  : a
                | b
                ;

            a   : (A B)? => <<p(LATEXT(2))>>? (A  B | D E)
                ;

            b   : <<q(LATEXT(2))>>? D E
                ;


      In this case, the sequence (D E) in rule "a" which lies behind
      the guard is used to suppress the predicate with context (D E)
      in rule b.

                while ( (LA(1)==A || LA(1)==D)
            #if 0

            Part (or all) of predicate with depth > 1 suppressed by alternative
                without predicate

            pred  <<  q(LATEXT(2))>>?
                              depth=k=2  rule b  line 11  t2.g
              tree context:
                (root = D
                   E
                )

            The token sequence which is suppressed: ( D E )
            The sequence of references which generate that sequence of tokens:

               1 to ab          r1/1       line 1     t2.g
               2 ab             ab/1       line 4     t2.g
               3 to a           ab/1       line 4     t2.g
               4 a              a/1        line 8     t2.g
               5 #token D       a/1        line 8     t2.g
               6 #token E       a/1        line 8     t2.g

            #endif
            &&
            #if 0

            pred  <<  p(LATEXT(2))>>?
                              depth=k=2  ("=>" guard)  rule a  line 8  t2.g
              tree context:
                (root = A
                   B
                )

            #endif

            (! ( LA(1)==A && LA(2)==B ) || p(LATEXT(2)) )  {
                ab();
                ...

#169. (Changed in MR13) Predicate test optimization for depth=1 predicates

      When the MR12 generated a test of a predicate which had depth 1
      it would use the depth >1 routines, resulting in correct but
      inefficient behavior.  In MR13, a bit test is used.

#168. (Changed in MR13) Token expressions in context guards

      The token expressions appearing in context guards such as:

            (A B)? => <<test(LT(1))>>?  someRule

      are computed during an early phase of antlr processing.  As
      a result, prior to MR13, complex expressions such as:

            ~B
            L..U
            ~L..U
            TokClassName
            ~TokClassName

      were not computed properly.  This resulted in incorrect
      context being computed for such expressions.

      In MR13 these context guards are verified for proper semantics
      in the initial phase and then re-evaluated after complex token
      expressions have been computed in order to produce the correct
      behavior.

      Reported by Arpad Beszedes (beszedes@inf.u-szeged.hu).

#167. (Changed in MR13) ~L..U

      Prior to MR13, the complement of a token range was
      not properly computed.

#166. (Changed in MR13) token expression L..U

      The token U was represented as an unsigned char, restricting
      the use of L..U to cases where U was assigned a token number
      less than 256.  This is corrected in MR13.

#165. (Changed in MR13) option -newAST

      To create ASTs from an ANTLRTokenPtr antlr usually calls
      "new AST(ANTLRTokenPtr)".  This option generates a call
      to "newAST(ANTLRTokenPtr)" instead.  This allows a user
      to define a parser member function to create an AST object.

      Similar changes for ASTBase::tmake and ASTBase::link were not
      thought necessary since they do not create AST objects, only
      use existing ones.

#164. (Changed in MR13) Unused variable _astp

      For many compilations, we have lived with warnings about
      the unused variable _astp.  It turns out that this varible
      can *never* be used because the code which references it was
      commented out.

      This investigation was sparked by a note from Erwin Achermann
      (erwin.achermann@switzerland.org).

#163. (Changed in MR13) Incorrect makefiles for testcpp examples

      All the examples in pccts/testcpp/* had incorrect definitions
      in the makefiles for the symbol "CCC".  Instead of CCC=CC they
      had CC=$(CCC).

      There was an additional problem in testcpp/1/test.g due to the
      change in ANTLRToken::getText() to a const member function
      (Item #137).

      Reported by Maurice Mass (maas@cuci.nl).

#162. (Changed in MR13) Combining #token with #tokdefs

      When it became possible to change the print-name of a
      #token (Item #148) it became useful to give a #token
      statement whose only purpose was to giving a print name
      to the #token.  Prior to this change this could not be
      combined with the #tokdefs feature.

#161. (Changed in MR13) Switch -gxt inhibits generation of tokens.h

#160. (Changed in MR13) Omissions in list of names for remap.h

      When a user selects the -gp option antlr creates a list
      of macros in remap.h to rename some of the standard
      antlr routines from zzXXX to userprefixXXX.

      There were number of omissions from the remap.h name
      list related to the new trace facility.  This was reported,
      along with a fix, by Bernie Solomon (bernard@ug.eds.com).

#159. (Changed in MR13) Violations of classic C rules

      There were a number of violations of classic C style in
      the distribution kit.  This was reported, along with fixes,
      by Bernie Solomon (bernard@ug.eds.com).

#158. (Changed in MR13) #header causes problem for pre-processors

      A user who runs the C pre-processor on antlr source suggested
      that another syntax be allowed.  With MR13 such directives
      such as #header, #pragma, etc. may be written as "\#header",
      "\#pragma", etc.  For escaping pre-processor directives inside
      a #header use something like the following:

            \#header
            <<
                \#include <stdio.h>
            >>

#157. (Fixed in MR13) empty error sets for rules with infinite recursion

      When the first set for a rule cannot be computed due to infinite
      left recursion and it is the only alternative for a block then
      the error set for the block would be empty.  This would result
      in a fatal error.

      Reported by Darin Creason (creason@genedax.com)

#156. (Changed in MR13) DLGLexerBase::getToken() now public

#155. (Changed in MR13) Context behind predicates can suppress

      With -mrhoist enabled the context behind a guarded predicate can
      be used to suppress other predicates.  Consider the following grammar:

        r0 : (r1)+;

        r1  : rp
            | rq
            ;
        rp  : <<p LATEXT(1)>>? B ;
        rq : (A)? => <<q LATEXT(1)>>? (A|B);

      In earlier versions both predicates "p" and "q" would be hoisted into
      rule r0. With MR12c predicate p is suppressed because the context which
      follows predicate q includes "B" which can "cover" predicate "p".  In
      other words, in trying to decide in r0 whether to call r1, it doesn't
      really matter whether p is false or true because, either way, there is
      a valid choice within r1.

#154. (Changed in MR13) Making hoist suppression explicit using <<nohoist>>

      A common error, even among experienced pccts users, is to code
      an init-action to inhibit hoisting rather than a leading action.
      An init-action does not inhibit hoisting.

      This was coded:

        rule1 : <<;>> rule2

      This is what was meant:

        rule1 : <<;>> <<;>> rule2

      With MR13, the user can code:

        rule1 : <<;>> <<nohoist>> rule2

      The following will give an error message:

        rule1 : <<nohoist>> rule2

      If the <<nohoist>> appears as an init-action rather than a leading
      action an error message is issued.  The meaning of an init-action
      containing "nohoist" is unclear: does it apply to just one
      alternative or to all alternatives ?








        -------------------------------------------------------
        Note:  Items #153 to #1 are now in a separate file named
                CHANGES_FROM_133_BEFORE_MR13.txt
        -------------------------------------------------------