Planet
navi homePPSaboutscreenshotsdownloaddevelopmentforum

source: downloads/boost_1_34_1/libs/tokenizer/tokenizerfunction.htm @ 29

Last change on this file since 29 was 29, checked in by landauf, 16 years ago

updated boost from 1_33_1 to 1_34_1
File size: 4.5 KB
Line 
1<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2
3<html>
4<head>
5  <meta http-equiv="Content-Language" content="en-us">
6  <meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
7  <meta name="GENERATOR" content="Microsoft FrontPage 6.0">
8  <meta name="ProgId" content="FrontPage.Editor.Document">
9
10  <title>TokenizerFunction Concept</title>
11</head>
12
13<body bgcolor="#FFFFFF" text="#000000" link="#0000EE" vlink="#551A8B" alink=
14"#FF0000">
15  <p><img src="../../boost.png" alt="C++ Boost" width="277" height="86"></p>
16
17  <h1 align="center">TokenizerFunction Concept</h1>
18
19  <p>A TokenizerFunction is a functor whose purpose is to parse a given
20  sequence until exactly 1 token has been found or the end is reached. It
21  then updates the token, and informs the caller of the location in the
22  sequence of the next element immediately after the last element of the
23  sequence that was parsed for the current token.</p>
24
25  <h2>Refinement of</h2>
26
27  <p>Assignable, CopyConstructable</p>
28
29  <h2>Notation</h2>
30
31  <table border="1" summary="">
32    <tr>
33      <td valign="top"><tt>X</tt></td>
34
35      <td valign="top">A type that is a model of TokenizerFunction</td>
36    </tr>
37
38    <tr>
39      <td valign="top"><tt>func</tt></td>
40
41      <td valign="top">Object of type <tt>X</tt></td>
42    </tr>
43
44    <tr>
45      <td valign="top"><tt>tok</tt></td>
46
47      <td valign="top">Object of Token</td>
48    </tr>
49
50    <tr>
51      <td>next</td>
52
53      <td>iterator that points to the first unparsed element of the sequence
54      being parsed</td>
55    </tr>
56
57    <tr>
58      <td>end</td>
59
60      <td>iterator that points to the past the end of the sequence being
61      parsed</td>
62    </tr>
63  </table>
64
65  <h2>Definitions</h2>
66
67  <p>A token is the result of parsing a sequence.</p>
68
69  <h2>Valid expressions</h2>
70
71  <p>In addition to the expression in Assignable and CopyConstructable the
72  following expressions are valid</p>
73
74  <table border="1" summary="">
75    <tr>
76      <th>Name</th>
77
78      <th>Expression</th>
79
80      <th>Return type</th>
81    </tr>
82
83    <tr>
84      <td valign="top">Functor</td>
85
86      <td valign="top"><tt>func(next, end, tok)</tt></td>
87
88      <td valign="top"><tt>bool</tt></td>
89    </tr>
90
91    <tr>
92      <td valign="top">reset</td>
93
94      <td valign="top"><tt>reset()</tt></td>
95
96      <td valign="top"><tt>void</tt></td>
97    </tr>
98  </table>
99
100  <h2>Expression semantics</h2>
101
102  <p>In addition to the expression semantics in Assignable and
103  CopyConstructable, TokenizerFunction has the following expression
104  semantcs</p>
105
106  <table border="1" summary="">
107    <tr>
108      <th>Name</th>
109
110      <th>Expression</th>
111
112      <th>Precondition</th>
113
114      <th>Semantics</th>
115
116      <th>Postcondition</th>
117    </tr>
118
119    <tr>
120      <td>operator()</td>
121
122      <td><tt>func(next, end, tok)</tt></td>
123
124      <td><tt>next</tt> and <tt>end</tt> are valid iterators to the same
125      sequence. next is a reference the function is free to modify. tok is
126      constructed.</td>
127
128      <td>The return value indicates whether a new token was found in the
129      sequence (next,end]</td>
130
131      <td>If the return value is true, the new token is assigned to tok. next
132      is always updated to the position where parsing should start on the
133      subsequent call.</td>
134    </tr>
135
136    <tr>
137      <td>reset</td>
138
139      <td><tt>reset()</tt></td>
140
141      <td><tt>None</tt></td>
142
143      <td>Clears out all state variables that are used by the object in
144      parsing the current sequence.</td>
145
146      <td>A new sequence to parse can be given.</td>
147    </tr>
148  </table>
149
150  <h2>Complexity guarantees</h2>
151
152  <p>No guarantees. Models of TokenizerFunction are free to define their own
153  complexity</p>
154
155  <h2>Models</h2>
156
157  <p><a href="escaped_list_separator.htm">escaped_list_separator</a></p>
158
159  <p><a href="offset_separator.htm">offset_separator</a></p>
160
161  <p><a href=
162  "char_delimiters_separator.htm">char_delimiters_separator</a></p>
163
164  <p>&nbsp;</p>
165  <hr>
166
167  <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src=
168  "http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional"
169  height="31" width="88"></a></p>
170
171  <p>Revised
172  <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25
173  December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p>
174
175  <p><i>Copyright &copy; 2001 John R. Bandela</i></p>
176
177  <p><i>Distributed under the Boost Software License, Version 1.0. (See
178  accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or
179  copy at <a href=
180  "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p>
181</body>
182</html>
Note: See TracBrowser for help on using the repository browser.