Updated conversion command line option with unicode support

[mirror_edk2.git] / EdkCompatibilityPkg / Other / Maintained / Tools / Pccts / CHANGES_FROM_133.txt

=======================================================================\r
List of Implemented Fixes and Changes for Maintenance Releases of PCCTS\r
\r
\r
 For a summary of the most significant changes see CHANGES_SUMMARY.TXT\r
\r
=======================================================================\r
\r
                               DISCLAIMER\r
\r
 The software and these notes are provided "as is".  They may include\r
 typographical or technical errors and their authors disclaims all\r
 liability of any kind or nature for damages due to error, fault,\r
 defect, or deficiency regardless of cause.  All warranties of any\r
 kind, either express or implied, including, but not limited to, the\r
 implied  warranties of merchantability and fitness for a particular\r
 purpose are disclaimed.\r
\r
\r
        -------------------------------------------------------\r
        Note:  Items #153 to #1 are now in a separate file named\r
                CHANGES_FROM_133_BEFORE_MR13.txt\r
        -------------------------------------------------------\r
\r
#261. (Changed in MR19) Defer token fetch for C++ mode\r
\r
    Item #216 has been revised to indicate that use of the defer fetch\r
    option (ZZDEFER_FETCH) requires dlg option -i.\r
\r
#260. (MR22) Raise default lex buffer size from 8,000 to 32,000 bytes.\r
\r
    ZZLEXBUFSIZE is the size (in bytes) of the buffer used by dlg \r
    generated lexers.  The default value has been raised to 32,000 and\r
    the value used by antlr, dlg, and sorcerer has also been raised to\r
    32,000.\r
\r
#259. (MR22) Default function arguments in C++ mode.\r
\r
    If a rule is declared:\r
\r
            rr [int i = 0] : ....\r
\r
    then the declaration generated by pccts resembles:\r
\r
            void rr(int i = 0);\r
\r
    however, the definition must omit the default argument:\r
\r
            void rr(int i) {...}\r
\r
    In the past the default value was not omitted.  In MR22\r
    the generated code resembles:\r
\r
            void rr(int i /* = 0 */ ) {...}\r
\r
    Implemented by Volker H. Simonis (simonis@informatik.uni-tuebingen.de)\r
\r
#258. (MR22)  Using a base class for your parser\r
\r
    In item #102 (MR10) the class statement was extended to allow one\r
    to specify a base class other than ANTLRParser for the generated\r
    parser.  It turned out that this was less than useful because\r
    the constructor still specified ANTLRParser as the base class.\r
\r
    The class statement now uses the first identifier appearing after\r
    the ":" as the name of the base class.  For example:\r
\r
        class MyParser : public FooParser {\r
\r
    Generates in MyParser.h:\r
\r
            class MyParser : public FooParser {\r
\r
    Generates in MyParser.cpp something that resembles:\r
\r
            MyParser::MyParser(ANTLRTokenBuffer *input) :\r
                                         FooParser(input,1,0,0,4)\r
            {\r
            	token_tbl = _token_tbl;\r
            	traceOptionValueDefault=1;		// MR10 turn trace ON\r
            }\r
\r
    The base class must constructor must have a signature similar to\r
    that of ANTLRParser.\r
\r
#257. (MR21a) Removed dlg statement that -i has no effect in C++ mode.\r
\r
    This was incorrect.\r
\r
#256. (MR21a) Malformed syntax graph causes crash after error message.\r
\r
    In the past, certain kinds of errors in the very first grammar\r
    element could cause the construction of a malformed graph \r
    representing the grammar.  This would eventually result in a\r
    fatal internal error.  The code has been changed to be more\r
    resistant to this particular error.\r
\r
#255. (MR21a) ParserBlackBox(FILE* f) \r
\r
    This constructor set openByBlackBox to the wrong value.\r
\r
    Reported by Kees Bakker (kees_bakker@tasking.nl).\r
\r
#254. (MR21a) Reporting syntax error at end-of-file\r
\r
    When there was a syntax error at the end-of-file the syntax\r
    error routine would substitute "<eof>" for the programmer's\r
    end-of-file symbol.  This substitution is now done only when\r
    the programmer does not define his own end-of-file symbol\r
    or the symbol begins with the character "@".\r
\r
    Reported by Kees Bakker (kees_bakker@tasking.nl).\r
\r
#253. (MR21) Generation of block preamble (-preamble and -preamble_first)\r
\r
    The antlr option -preamble causes antlr to insert the code\r
    BLOCK_PREAMBLE at the start of each rule and block.  It does\r
    not insert code before rules references, token references, or\r
    actions.  By properly defining the macro BLOCK_PREAMBLE the\r
    user can generate code which is specific to the start of blocks.\r
\r
    The antlr option -preamble_first is similar, but inserts the\r
    code BLOCK_PREAMBLE_FIRST(PreambleFirst_123) where the symbol\r
    PreambleFirst_123 is equivalent to the first set defined by\r
    the #FirstSetSymbol described in Item #248.\r
\r
    I have not investigated how these options interact with guess\r
    mode (syntactic predicates).\r
\r
#252. (MR21) Check for null pointer in trace routine\r
\r
    When some trace options are used when the parser is generated\r
    without the trace enabled, the current rule name may be a\r
    NULL pointer.  A guard was added to check for this in\r
    restoreState.\r
\r
    Reported by Douglas E. Forester (dougf@projtech.com).\r
\r
#251. (MR21) Changes to #define zzTRACE_RULES\r
\r
    The macro zzTRACE_RULES was being use to pass information to\r
    AParser.h.  If this preprocessor symbol was not properly\r
    set the first time AParser.h was #included, the declaration\r
    of zzTRACEdata would be omitted (it is used by the -gd option).\r
    Subsequent #includes of AParser.h would be skipped because of \r
    the #ifdef guard, so the declaration of zzTracePrevRuleName would\r
    never be made.  The result was that proper compilation was very \r
    order dependent.\r
\r
    The declaration of zzTRACEdata was made unconditional and the\r
    problem of removing unused declarations will be left to optimizers.\r
    \r
    Diagnosed by Douglas E. Forester (dougf@projtech.com).\r
\r
#250. (MR21) Option for EXPERIMENTAL change to error sets for blocks\r
\r
    The antlr option -mrblkerr turns on an experimental feature\r
    which is supposed to provide more accurate syntax error messages\r
    for k=1, ck=1 grammars.  When used with k>1 or ck>1 grammars the\r
    behavior should be no worse than the current behavior.\r
\r
    There is no problem with the matching of elements or the computation\r
    of prediction expressions in pccts.  The task is only one of listing\r
    the most appropriate tokens in the error message.  The error sets used\r
    in pccts error messages are approximations of the exact error set when\r
    optional elements in (...)* or (...)+ are involved.  While entirely\r
    correct, the error messages are sometimes not 100% accurate.  \r
\r
    There is also a minor philosophical issue.  For example, suppose the\r
    grammar expects the token to be an optional A followed by Z, and it \r
    is X.  X, of course, is neither A nor Z, so an error message is appropriate.\r
    Is it appropriate to say "Expected Z" ?  It is correct, it is accurate,\r
    but it is not complete.  \r
\r
    When k>1 or ck>1 the problem of providing the exactly correct\r
    list of tokens for the syntax error messages ends up becoming\r
    equivalent to evaluating the prediction expression for the\r
    alternatives twice. However, for k=1 ck=1 grammars the prediction\r
    expression can be computed easily and evaluated cheaply, so I\r
    decided to try implementing it to satisfy a particular application.\r
    This application uses the error set in an interactive command language\r
    to provide prompts which list the alternatives available at that\r
    point in the parser.  The user can then enter additional tokens to\r
    complete the command line.  To do this required more accurate error \r
    sets then previously provided by pccts.\r
\r
    In some cases the default pccts behavior may lead to more robust error\r
    recovery or clearer error messages then having the exact set of tokens.\r
    This is because (a) features like -ge allow the use of symbolic names for\r
    certain sets of tokens, so having extra tokens may simply obscure things\r
    and (b) the error set is use to resynchronize the parser, so a good\r
    choice is sometimes more important than having the exact set.\r
\r
    Consider the following example:\r
\r
            Note:  All examples code has been abbreviated\r
            to the absolute minimum in order to make the\r
            examples concise.\r
\r
        star1 : (A)* Z;\r
\r
    The generated code resembles:\r
\r
           old                new (with -mrblkerr)\r
        -------------         --------------------\r
        for (;;) {            for (;;) {\r
            match(A);           match(A);\r
        }                     }\r
        match(Z);             if (! A and ! Z) then\r
                                FAIL(...{A,Z}...);\r
                              }\r
                              match(Z);\r
\r
\r
        With input X\r
            old message: Found X, expected Z\r
            new message: Found X, expected A, Z\r
\r
    For the example:\r
\r
        star2 : (A|B)* Z;\r
\r
           old                      new (with -mrblkerr)\r
        -------------               --------------------\r
        for (;;) {                  for (;;) {\r
          if (!A and !B) break;       if (!A and !B) break;\r
          if (...) {                  if (...) {\r
            <same ...>                  <same ...>\r
          }                           }\r
          else {                      else {\r
            FAIL(...{A,B,Z}...)         FAIL(...{A,B}...);\r
          }                           }\r
        }                           }\r
        match(B);                   if (! A and ! B and !Z) then\r
                                        FAIL(...{A,B,Z}...);\r
                                    }\r
                                    match(B);\r
\r
        With input X\r
            old message: Found X, expected Z\r
            new message: Found X, expected A, B, Z\r
        With input A X\r
            old message: Found X, expected Z\r
            new message: Found X, expected A, B, Z\r
\r
            This includes the choice of looping back to the\r
            star block.\r
\r
    The code for plus blocks:\r
\r
        plus1 : (A)+ Z;\r
\r
    The generated code resembles:\r
\r
           old                  new (with -mrblkerr)\r
        -------------           --------------------\r
        do {                    do {\r
          match(A);               match(A);\r
        } while (A)             } while (A)\r
        match(Z);               if (! A and ! Z) then\r
                                  FAIL(...{A,Z}...);\r
                                }\r
                                match(Z);\r
\r
        With input A X\r
            old message: Found X, expected Z\r
            new message: Found X, expected A, Z\r
\r
            This includes the choice of looping back to the\r
            plus block.\r
\r
    For the example:\r
\r
        plus2 : (A|B)+ Z;\r
\r
           old                    new (with -mrblkerr)\r
        -------------             --------------------\r
        do {                        do {\r
          if (A) {                    <same>\r
            match(A);                 <same>\r
          } else if (B) {             <same>\r
            match(B);                 <same>\r
          } else {                    <same>\r
            if (cnt > 1) break;       <same>\r
            FAIL(...{A,B,Z}...)         FAIL(...{A,B}...);\r
          }                           }\r
          cnt++;                      <same>\r
        }                           }\r
\r
        match(Z);                   if (! A and ! B and !Z) then\r
                                        FAIL(...{A,B,Z}...);\r
                                    }\r
                                    match(B);\r
\r
        With input X\r
            old message: Found X, expected A, B, Z\r
            new message: Found X, expected A, B\r
        With input A X\r
            old message: Found X, expected Z\r
            new message: Found X, expected A, B, Z\r
\r
            This includes the choice of looping back to the\r
            star block.\r
    \r
#249. (MR21) Changes for DEC/VMS systems\r
\r