| 1 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
|---|
| 2 | <html> |
|---|
| 3 | <head> |
|---|
| 4 | <title>The Lexer Interface</title> |
|---|
| 5 | <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> |
|---|
| 6 | <link href="theme/style.css" rel="stylesheet" type="text/css"> |
|---|
| 7 | </head> |
|---|
| 8 | |
|---|
| 9 | <body> |
|---|
| 10 | <table width="100%" border="0" cellspacing="2" background="theme/bkd2.gif"> |
|---|
| 11 | <tr> |
|---|
| 12 | <td width="21"> <h1></h1></td> |
|---|
| 13 | <td width="885"> <font face="Verdana, Arial, Helvetica, sans-serif"><b><font size="6">The |
|---|
| 14 | Lexer Iterator Interface</font></b></font></td> |
|---|
| 15 | <td width="96"><a href="http://www.boost.org"><img src="theme/wave.gif" width="93" height="68" align="right" border="0"></a></td> |
|---|
| 16 | </tr> |
|---|
| 17 | </table> |
|---|
| 18 | <br> |
|---|
| 19 | <table border="0"> |
|---|
| 20 | <tr> |
|---|
| 21 | <td width="10"></td> |
|---|
| 22 | <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td> |
|---|
| 23 | <td width="30"><a href="class_reference_ctxpolicy.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td> |
|---|
| 24 | <td width="30"><a href="class_reference_tokentype.html"><img src="theme/r_arr.gif" border="0"></a></td> |
|---|
| 25 | </tr> |
|---|
| 26 | </table> |
|---|
| 27 | <blockquote> |
|---|
| 28 | <p><a href="class_reference_lexer.html#introduction">Introduction</a><br> |
|---|
| 29 | <a href="class_reference_lexer.html#header_synopsis">Wave Lexer |
|---|
| 30 | synopsis</a><br> |
|---|
| 31 | <a href="class_reference_lexer.html#public_typedefs">Public Typedefs</a><br> |
|---|
| 32 | <a href="class_reference_lexer.html#member_functions">Member functions</a></p> |
|---|
| 33 | </blockquote> |
|---|
| 34 | <h2><b><a name="introduction"></a>Introduction</b></h2> |
|---|
| 35 | <p>Every lexer, which should be used in conjunction with the <tt>Wave</tt> library, has to return tokens formed from the input stream. These tokens should conform to the synopsis described in the topic <a href="class_reference_tokentype.html">The Token Type</a>. The lexer type should expose an interface which conforms at least to a <tt>forward_iterator</tt> (in the sense defined by the the C++ Standard) returning the token type. The code sample below does not show the definition of this forward iterator interface because this is highly implementation defined.</p> |
|---|
| 36 | <h2><a name="header_synopsis"></a>Wave lexer |
|---|
| 37 | synopsis (header: <a href="http://cvs.sourceforge.net/viewcvs.py/boost/boost/boost/wave/cpplexer/cpp_lex_interface.hpp?view=markup">wave/cpplexer/cpp_lexer_interface.hpp</a>)</h2> |
|---|
| 38 | <pre> <span class="keyword">struct</span> lex_iterator |
|---|
| 39 | { |
|---|
| 40 | <span class="keyword">typedef</span> boost::wave::lex_token<> <a href="class_reference_lexer.html#public_typedefs">token_type</a>; |
|---|
| 41 | |
|---|
| 42 | <span class="comment"> // Every lex_iterator should implement at least a complete |
|---|
| 43 | // forward_iterator interface (not shown here) |
|---|
| 44 | </span><span class="keyword"> typedef</span> std::forward_iterator_tag iterator_category; |
|---|
| 45 | |
|---|
| 46 | <span class="comment"> // additional requirements |
|---|
| 47 | </span> <a href="class_reference_lexer.html#member_functions">lex_iterator</a>(); |
|---|
| 48 | |
|---|
| 49 | <span class="keyword">template</span> <<span class="keyword">typename</span> Iterator> |
|---|
| 50 | <a href="class_reference_lexer.html#constructor">lex_iterator</a>(Iterator <span class="keyword">const</span> &first, Iterator <span class="keyword">const</span> &last |
|---|
| 51 | <span class="keyword">typename</span> token_type::position_type <span class="keyword">const</span> &pos, |
|---|
| 52 | boost::wave::language_support language) |
|---|
| 53 | }; |
|---|
| 54 | |
|---|
| 55 | </pre> |
|---|
| 56 | <p>Please note, that the <tt>lex_iterator</tt> defined in the library header <a href="http://cvs.sourceforge.net/viewcvs.py/boost/boost/boost/wave/cpplexer/cpp_lex_interface.hpp?view=markup">wave/cpplexer/cpp_lexer_interface.hpp</a> actually is a template class taking the token type to use as its template parameter. This is omitted in the synopsis above because it is an implementation detail of the Re2C lexer provided as part of the Wave library.</p> |
|---|
| 57 | <p>If you want to use Wave in conjunction with your own lexing component this will have to conform to the interface described above only. </p> |
|---|
| 58 | <h2><a name="public_typedefs" id="public_typedefs"></a>Public Typedefs</h2> |
|---|
| 59 | <p>Besides the typedefs mandated for a <tt>forward_iterator</tt> by the C++ standard every lexer to be used with the <tt>Wave</tt> library should define the following typedefs: </p> |
|---|
| 60 | <table width="90%" border="0" align="center"> |
|---|
| 61 | <tr> |
|---|
| 62 | <td colspan="2" class="table_title"><b>Public typedef's defined by the <tt>boost::wave::context</tt> class </b></td> |
|---|
| 63 | </tr> |
|---|
| 64 | <tr> |
|---|
| 65 | <td width="32%" class="table_cells"><code>token_type</code></td> |
|---|
| 66 | <td width="68%" class="table_cells"><p>The <tt>token</tt> type returned by the lexer. This is type is used as the return value of the main iterators provided by the <tt>boost::wave::context</tt> object too. </p></td> |
|---|
| 67 | </tr> |
|---|
| 68 | </table> |
|---|
| 69 | <h2><a name="member_functions"></a>Member functions</h2> |
|---|
| 70 | <p>Besides the functions, which should be provided for <tt>forward_iterators</tt> as mandated by the C++ Standard, every lexer must implement the following functions to be used with the <tt>Wave</tt> library: </p> |
|---|
| 71 | <p><a name="constructor" id="constructor"></a><b>constructor</b></p> |
|---|
| 72 | <pre> lex_iterator();</pre> |
|---|
| 73 | <blockquote> |
|---|
| 74 | <p>The default constructor should construct a lexer iterator, which may be used as the end iterator of the provided iterator range.</p> |
|---|
| 75 | </blockquote> |
|---|
| 76 | <pre> |
|---|
| 77 | <span class="keyword">template</span> <<span class="keyword">typename</span> Iterator> |
|---|
| 78 | lex_iterator(Iterator <span class="keyword">const</span> &first, Iterator <span class="keyword">const</span> &last, |
|---|
| 79 | <span class="keyword">typename</span> token_type::position_type <span class="keyword">const</span> &pos, |
|---|
| 80 | boost::wave::language_support language);</pre> |
|---|
| 81 | <blockquote> |
|---|
| 82 | <p>The second constructor should construct a lexer iterator, which may be used as a iterator traversing over the token sequence, generated by the lexer class.</p> |
|---|
| 83 | <p>The pair of iterators <tt>first</tt> and <tt>last</tt> should represent the input stream to be tokenized by the given lexer class. </p> |
|---|
| 84 | <p>The parameter <tt>pos</tt> contains the initial position information to be used for token generation.</p> |
|---|
| 85 | <p>The parameter <tt>language</tt> controls the reuqired mode with which the lexer should be initialised. </p> |
|---|
| 86 | </blockquote> |
|---|
| 87 | <table border="0"> |
|---|
| 88 | <tr> |
|---|
| 89 | <td width="10"></td> |
|---|
| 90 | <td width="30"><a href="../index.html"><img src="theme/u_arr.gif" border="0"></a></td> |
|---|
| 91 | <td width="30"><a href="class_reference_ctxpolicy.html"><img src="theme/l_arr.gif" width="20" height="19" border="0"></a></td> |
|---|
| 92 | <td width="30"><a href="class_reference_tokentype.html"><img src="theme/r_arr.gif" border="0"></a></td> |
|---|
| 93 | </tr> |
|---|
| 94 | </table> |
|---|
| 95 | <hr size="1"> |
|---|
| 96 | <p class="copyright">Copyright © 2003-2007 Hartmut Kaiser<br> |
|---|
| 97 | <br> |
|---|
| 98 | <font size="2">Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) </font> </p> |
|---|
| 99 | <p class="copyright"><span class="updated">Last updated: |
|---|
| 100 | <!-- #BeginDate format:fcAm1m -->Thursday, December 1, 2005 11:53<!-- #EndDate --> |
|---|
| 101 | </span> |
|---|
| 102 | </p> |
|---|
| 103 | </body> |
|---|
| 104 | </html> |
|---|
| 105 | |
|---|
