projects / ceph.git / blame
| Commit | Line | Data |
|---|---|---|
| 7c673cae FG |
1 | [/============================================================================== |
| 2 | Copyright (C) 2001-2011 Joel de Guzman | |
| 3 | Copyright (C) 2001-2011 Hartmut Kaiser | |
| 4 | ||
| 5 | Distributed under the Boost Software License, Version 1.0. (See accompanying | |
| 6 | file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) | |
| 7 | ===============================================================================/] | |
| 8 | ||
| 9 | [section:lexer_tutorials __lex__ Tutorials Overview] | |
| 10 | ||
| 11 | The __lex__ library implements several components on top of possibly different | |
| 12 | lexer generator libraries. It exposes a pair of iterators, which, when | |
| 13 | dereferenced, return a stream of tokens generated from the underlying character | |
| 14 | stream. The generated tokens are based on the token definitions supplied by the | |
| 15 | user. | |
| 16 | ||
| 17 | Currently, __lex__ is built on top of Ben Hanson's excellent __lexertl__ | |
| 18 | library (which is a proposed Boost library). __lexertl__ provides the necessary | |
| 19 | functionality to build state machines based on a set of supplied regular | |
| 20 | expressions. But __lex__ is not restricted to be used with __lexertl__. We | |
| 21 | expect it to be usable in conjunction with any other lexical scanner generator | |
| 22 | library, all what needs to be implemented is a set of wrapper objects exposing a | |
| 23 | well defined interface as described in this documentation. | |
| 24 | ||
| 25 | [note For the sake of clarity all examples in this documentation assume | |
| 26 | __lex__ to be used on top of __lexertl__.] | |
| 27 | ||
| 28 | Building a lexer using __lex__ is highly configurable, where most of this | |
| 29 | configuration is done at compile time. Almost all of the configurable | |
| 30 | parameters have generally useful default values, allowing project startup to be | |
| 31 | a easy and straightforward task. Here is a (non-complete) list of features you | |
| 32 | can tweak to adjust the generated lexer instance to the actual needs: | |
| 33 | ||
| 34 | * Select and customize the token type to be generated by the lexer instance. | |
| 35 | * Select and customize the token value types the generated token instances will | |
| 36 | be able to hold. | |
| 37 | * Select the iterator type of the underlying input stream, which will be used | |
| 38 | as the source for the character stream to tokenize. | |
| 39 | * Customize the iterator type returned by the lexer to enable debug support, | |
| 40 | special handling of certain input sequences, etc. | |
| 41 | * Select the /dynamic/ or the /static/ runtime model for the lexical | |
| 42 | analyzer. | |
| 43 | ||
| 44 | Special care has been taken during the development of the library that | |
| 45 | optimal code will be generated regardless of the configuration options | |
| 46 | selected. | |
| 47 | ||
| 48 | The series of tutorial examples of this section will guide you through some | |
| 49 | common use cases helping to understand the big picture. The first two quick | |
| 50 | start examples (__sec_lex_quickstart_1__ and __sec_lex_quickstart_2__) | |
| 51 | introduce the __lex__ library while building two stand alone applications, not | |
| 52 | being connected to or depending on any other part of __spirit__. The section | |
| 53 | __sec_lex_quickstart_3__ demonstrates how to use a lexer in conjunction with a | |
| 54 | parser (where obviously the parser is built using __qi__). | |
| 55 | ||
| 56 | [endsect] | |
| 57 |