]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | [/============================================================================== |
2 | Copyright (C) 2001-2011 Joel de Guzman | |
3 | Copyright (C) 2001-2011 Hartmut Kaiser | |
4 | ||
5 | Distributed under the Boost Software License, Version 1.0. (See accompanying | |
6 | file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) | |
7 | ===============================================================================/] | |
8 | [section:lexer_api Lexer API] | |
9 | ||
10 | [heading Description] | |
11 | ||
12 | The library provides a couple of free functions to make using the lexer a snap. | |
13 | These functions have three forms. The first form, `tokenize`, simplifies the | |
14 | usage of a stand alone lexer (without parsing). The second form, | |
15 | `tokenize_and_parse`, combines a lexer step with parsing on | |
16 | the token level (without a skipper). The third, `tokenize_and_phrase_parse`, | |
17 | works on the token level as well, but additionally employs a skip parser. The | |
18 | latter two versions can take in attributes by reference that will hold the | |
19 | parsed values on a successful parse. | |
20 | ||
21 | [heading Header] | |
22 | ||
23 | // forwards to <boost/spirit/home/lex/tokenize_and_parse.hpp> | |
24 | #include <boost/spirit/include/lex_tokenize_and_parse.hpp> | |
25 | ||
26 | For variadic attributes: | |
27 | ||
28 | // forwards to <boost/spirit/home/lex/tokenize_and_parse_attr.hpp> | |
29 | #include <boost/spirit/include/lex_tokenize_and_parse_attr.hpp> | |
30 | ||
31 | The variadic attributes version of the API allows one or more | |
32 | attributes to be passed into the API functions. The functions taking two | |
33 | or more attributes are usable when the parser expression is a | |
34 | __qi_sequence__ only. In this case each of the | |
35 | attributes passed have to match the corresponding part of the sequence. | |
36 | ||
37 | Also, see __include_structure__. | |
38 | ||
39 | [heading Namespace] | |
40 | ||
41 | [table | |
42 | [[Name]] | |
43 | [[`boost::spirit::lex::tokenize` ]] | |
44 | [[`boost::spirit::lex::tokenize_and_parse` ]] | |
45 | [[`boost::spirit::lex::tokenize_and_phrase_parse` ]] | |
46 | [[`boost::spirit::qi::skip_flag::postskip` ]] | |
47 | [[`boost::spirit::qi::skip_flag::dont_postskip` ]] | |
48 | ] | |
49 | ||
50 | [heading Synopsis] | |
51 | ||
52 | The `tokenize` function is one of the main lexer API functions. It | |
53 | simplifies using a lexer to tokenize a given input sequence. It's main | |
54 | purpose is to use the lexer to tokenize all the input. | |
55 | ||
56 | Both functions take a pair of iterators spanning the underlying input | |
57 | stream to scan, the lexer object (built from the token definitions), | |
58 | and an (optional) functor being called for each of the generated tokens. If no | |
59 | function object `f` is given, the generated tokens will be discarded. | |
60 | ||
61 | The functions return `true` if the scanning of the input succeeded (the | |
62 | given input sequence has been successfully matched by the given | |
63 | token definitions). | |
64 | ||
65 | The argument `f` is expected to be a function (callable) object taking a single | |
66 | argument of the token type and returning a bool, indicating whether | |
67 | the tokenization should be canceled. If it returns `false` the function | |
68 | `tokenize` will return `false` as well. | |
69 | ||
70 | The `initial_state` argument forces lexing to start with the given lexer state. | |
71 | If this is omitted lexing starts in the `"INITIAL"` state. | |
72 | ||
73 | template <typename Iterator, typename Lexer> | |
74 | inline bool | |
75 | tokenize( | |
76 | Iterator& first | |
77 | , Iterator last | |
78 | , Lexer const& lex | |
79 | , typename Lexer::char_type const* initial_state = 0); | |
80 | ||
81 | template <typename Iterator, typename Lexer, typename F> | |
82 | inline bool | |
83 | tokenize( | |
84 | Iterator& first | |
85 | , Iterator last | |
86 | , Lexer const& lex | |
87 | , F f | |
88 | , typename Lexer::char_type const* initial_state = 0); | |
89 | ||
90 | The `tokenize_and_parse` function is one of the main lexer API | |
91 | functions. It simplifies using a lexer as the underlying token source | |
92 | while parsing a given input sequence. | |
93 | ||
94 | The functions take a pair of iterators spanning the underlying input | |
95 | stream to parse, the lexer object (built from the token definitions) | |
96 | and a parser object (built from the parser grammar definition). Additionally | |
97 | they may take the attributes for the parser step. | |
98 | ||
99 | The function returns `true` if the parsing succeeded (the given input | |
100 | sequence has been successfully matched by the given grammar). | |
101 | ||
102 | template <typename Iterator, typename Lexer, typename ParserExpr> | |
103 | inline bool | |
104 | tokenize_and_parse( | |
105 | Iterator& first | |
106 | , Iterator last | |
107 | , Lexer const& lex | |
108 | , ParserExpr const& expr) | |
109 | ||
110 | template <typename Iterator, typename Lexer, typename ParserExpr | |
111 | , typename Attr1, typename Attr2, ..., typename AttrN> | |
112 | inline bool | |
113 | tokenize_and_parse( | |
114 | Iterator& first | |
115 | , Iterator last | |
116 | , Lexer const& lex | |
117 | , ParserExpr const& expr | |
118 | , Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN); | |
119 | ||
120 | The functions `tokenize_and_phrase_parse` take a pair of iterators spanning | |
121 | the underlying input stream to parse, the lexer object (built from the token | |
122 | definitions) and a parser object (built from the parser grammar definition). | |
123 | The additional skipper parameter will be used as the skip parser during | |
124 | the parsing process. Additionally they may take the attributes for the parser | |
125 | step. | |
126 | ||
127 | The function returns `true` if the parsing succeeded (the given input | |
128 | sequence has been successfully matched by the given grammar). | |
129 | ||
130 | template <typename Iterator, typename Lexer, typename ParserExpr | |
131 | , typename Skipper> | |
132 | inline bool | |
133 | tokenize_and_phrase_parse( | |
134 | Iterator& first | |
135 | , Iterator last | |
136 | , Lexer const& lex | |
137 | , ParserExpr const& expr | |
138 | , Skipper const& skipper | |
139 | , BOOST_SCOPED_ENUM(skip_flag) post_skip = skip_flag::postskip); | |
140 | ||
141 | template <typename Iterator, typename Lexer, typename ParserExpr | |
142 | , typename Skipper, typename Attr1, typename Attr2, ..., typename AttrN> | |
143 | inline bool | |
144 | tokenize_and_phrase_parse( | |
145 | Iterator& first | |
146 | , Iterator last | |
147 | , Lexer const& lex | |
148 | , ParserExpr const& expr | |
149 | , Skipper const& skipper | |
150 | , Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN); | |
151 | ||
152 | template <typename Iterator, typename Lexer, typename ParserExpr | |
153 | , typename Skipper, typename Attr1, typename Attr2, ..., typename AttrN> | |
154 | inline bool | |
155 | tokenize_and_phrase_parse( | |
156 | Iterator& first | |
157 | , Iterator last | |
158 | , Lexer const& lex | |
159 | , ParserExpr const& expr | |
160 | , Skipper const& skipper | |
161 | , BOOST_SCOPED_ENUM(skip_flag) post_skip | |
162 | , Attr1 const& attr1, Attr2 const& attr2, ..., AttrN const& attrN); | |
163 | ||
164 | The maximum number of supported arguments is limited by the preprocessor | |
165 | constant `SPIRIT_ARGUMENTS_LIMIT`. This constant defaults to the value defined | |
166 | by the preprocessor constant `PHOENIX_LIMIT` (which in turn defaults to `10`). | |
167 | ||
168 | [note The variadic function with two or more attributes internally combine | |
169 | references to all passed attributes into a `fusion::vector` and forward | |
170 | this as a combined attribute to the corresponding one attribute function.] | |
171 | ||
172 | The `tokenize_and_phrase_parse` functions not taking an explicit `skip_flag` | |
173 | as one of their arguments invoke the passed skipper after a successful match | |
174 | of the parser expression. This can be inhibited by using the other versions of | |
175 | that function while passing `skip_flag::dont_postskip` to the corresponding | |
176 | argument. | |
177 | ||
178 | [heading Template parameters] | |
179 | ||
180 | [table | |
181 | [[Parameter] [Description]] | |
182 | [[`Iterator`] [__fwditer__ pointing to the underlying input sequence to parse.]] | |
183 | [[`Lexer`] [A lexer (token definition) object.]] | |
184 | [[`F`] [A function object called for each generated token.]] | |
185 | [[`ParserExpr`] [An expression that can be converted to a Qi parser.]] | |
186 | [[`Skipper`] [Parser used to skip white spaces.]] | |
187 | [[`Attr1`, `Attr2`, ..., `AttrN`][One or more attributes.]] | |
188 | ] | |
189 | ||
190 | [endsect] |