A TokenizerFunction is a functor whose purpose is to parse a given sequence until exactly 1 token has been found or the end is reached. It then updates the token, and informs the caller of the location in the sequence of the next element immediately after the last element of the sequence that was parsed for the current token.
Assignable, CopyConstructable
| X | A type that is a model of TokenizerFunction |
| func | Object of type X |
| tok | Object of Token |
| next | iterator that points to the first unparsed element of the sequence being parsed |
| end | iterator that points to the past the end of the sequence being parsed |
A token is the result of parsing a sequence.
In addition to the expression in Assignable and CopyConstructable the following expressions are valid
| Name | Expression | Return type |
|---|---|---|
| Functor | func(next, end, tok) | bool |
| reset | reset() | void |
In addition to the expression semantics in Assignable and CopyConstructable, TokenizerFunction has the following expression semantcs
| Name | Expression | Precondition | Semantics | Postcondition |
|---|---|---|---|---|
| operator() | func(next, end, tok) | next and end are valid iterators to the same sequence. next is a reference the function is free to modify. tok is constructed. | The return value indicates whether a new token was found in the sequence (next,end] | If the return value is true, the new token is assigned to tok. next is always updated to the position where parsing should start on the subsequent call. |
| reset | reset() | None | Clears out all state variables that are used by the object in parsing the current sequence. | A new sequence to parse can be given. |
No guarantees. Models of TokenizerFunction are free to define their own complexity
Revised 25 December, 2006
Copyright © 2001 John R. Bandela
Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)