ceph/src/boost/libs/spirit/doc/lex/lexer_tutorials.qbk

   1 [/==============================================================================
   2     Copyright (C) 2001-2011 Joel de Guzman
   3     Copyright (C) 2001-2011 Hartmut Kaiser
   4
   5     Distributed under the Boost Software License, Version 1.0. (See accompanying
   6     file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
   7 ===============================================================================/]
   8
   9 [section:lexer_tutorials __lex__ Tutorials Overview]
  10
  11 The __lex__ library implements several components on top of possibly different
  12 lexer generator libraries. It exposes a pair of iterators, which, when
  13 dereferenced, return a stream of tokens generated from the underlying character
  14 stream. The generated tokens are based on the token definitions supplied by the
  15 user.
  16
  17 Currently, __lex__ is built on top of Ben Hanson's excellent __lexertl__
  18 library (which is a proposed Boost library). __lexertl__ provides the necessary
  19 functionality to build state machines based on a set of supplied regular
  20 expressions. But __lex__ is not restricted to be used with __lexertl__. We
  21 expect it to be usable in conjunction with any other lexical scanner generator
  22 library, all what needs to be implemented is a set of wrapper objects exposing a
  23 well defined interface as described in this documentation.
  24
  25 [note   For the sake of clarity all examples in this documentation assume
  26         __lex__ to be used on top of __lexertl__.]
  27
  28 Building a lexer using __lex__ is highly configurable, where most of this
  29 configuration is done at compile time. Almost all of the configurable
  30 parameters have generally useful default values, allowing project startup to be
  31 a easy and straightforward task. Here is a (non-complete) list of features you
  32 can tweak to adjust the generated lexer instance to the actual needs:
  33
  34 * Select and customize the token type to be generated by the lexer instance.
  35 * Select and customize the token value types the generated token instances will
  36   be able to hold.
  37 * Select the iterator type of the underlying input stream, which will be used
  38   as the source for the character stream to tokenize.
  39 * Customize the iterator type returned by the lexer to enable debug support,
  40   special handling of certain input sequences, etc.
  41 * Select the /dynamic/ or the /static/ runtime model for the lexical
  42   analyzer.
  43
  44 Special care has been taken during the development of the library that
  45 optimal code will be generated regardless of the configuration options
  46 selected.
  47
  48 The series of tutorial examples of this section will guide you through some
  49 common use cases helping to understand the big picture. The first two quick
  50 start examples (__sec_lex_quickstart_1__ and __sec_lex_quickstart_2__)
  51 introduce the __lex__ library while building two stand alone applications, not
  52 being connected to or depending on any other part of __spirit__. The section
  53 __sec_lex_quickstart_3__ demonstrates how to use a lexer in conjunction with a
  54 parser (where obviously the parser is built using __qi__).
  55
  56 [endsect]
  57