| 1 | <html> |
|---|
| 2 | |
|---|
| 3 | <head> |
|---|
| 4 | <meta http-equiv="Content-Type" |
|---|
| 5 | content="text/html; charset=iso-8859-1"> |
|---|
| 6 | <meta name="GENERATOR" content="Microsoft FrontPage Express 2.0"> |
|---|
| 7 | <title>Introduction</title> |
|---|
| 8 | </head> |
|---|
| 9 | |
|---|
| 10 | <body bgcolor="#FFFFFF"> |
|---|
| 11 | |
|---|
| 12 | <p><img src="../../boost.png" alt="C++ Boost" width="277" |
|---|
| 13 | height="86"> <br> |
|---|
| 14 | </p> |
|---|
| 15 | |
|---|
| 16 | <h1 align="center">Introduction</h1> |
|---|
| 17 | |
|---|
| 18 | <p align="left">The boost Tokenizer package provides a flexible |
|---|
| 19 | and easy to use way to break of a string or other character |
|---|
| 20 | sequence into a series of tokens. Below is a simple example that |
|---|
| 21 | will break up a phrase into words.</p> |
|---|
| 22 | <div align="left"> |
|---|
| 23 | |
|---|
| 24 | <pre>// simple_example_1.cpp |
|---|
| 25 | #include<iostream> |
|---|
| 26 | #include<boost/tokenizer.hpp> |
|---|
| 27 | #include<string> |
|---|
| 28 | |
|---|
| 29 | int main(){ |
|---|
| 30 | using namespace std; |
|---|
| 31 | using namespace boost; |
|---|
| 32 | string s = "This is, a test"; |
|---|
| 33 | tokenizer<> tok(s); |
|---|
| 34 | for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){ |
|---|
| 35 | cout << *beg << "\n"; |
|---|
| 36 | } |
|---|
| 37 | }</pre> |
|---|
| 38 | </div> |
|---|
| 39 | |
|---|
| 40 | <p align="left">You can choose how the string gets broken up. You |
|---|
| 41 | do this by specifying the TokenizerFunction. If you do not |
|---|
| 42 | specify anything, the default TokenizerFunction is |
|---|
| 43 | char_delimiters_separator<char> which defaults to breaking |
|---|
| 44 | up a string based on space and punctuation. Here is an example of |
|---|
| 45 | using another TokenizerFunction called escaped_list_separator. |
|---|
| 46 | This TokenizerFunction parses a superset of comma separated value |
|---|
| 47 | (csv) lines. The format looks like this</p> |
|---|
| 48 | |
|---|
| 49 | <p align="left">Field 1,"putting quotes around fields, |
|---|
| 50 | allows commas",Field 3</p> |
|---|
| 51 | |
|---|
| 52 | <p align="left">Below is an example that will break the previous |
|---|
| 53 | line into its 3 fields</p> |
|---|
| 54 | <div align="left"> |
|---|
| 55 | |
|---|
| 56 | <pre>// simple_example_2.cpp |
|---|
| 57 | #include<iostream> |
|---|
| 58 | #include<boost/tokenizer.hpp> |
|---|
| 59 | #include<string> |
|---|
| 60 | |
|---|
| 61 | int main(){ |
|---|
| 62 | using namespace std; |
|---|
| 63 | using namespace boost; |
|---|
| 64 | string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3"; |
|---|
| 65 | tokenizer<escaped_list_separator<char> > tok(s); |
|---|
| 66 | for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){ |
|---|
| 67 | cout << *beg << "\n"; |
|---|
| 68 | } |
|---|
| 69 | }</pre> |
|---|
| 70 | </div> |
|---|
| 71 | |
|---|
| 72 | <p align="left">Finally, for some TokenizerFunctions you have to |
|---|
| 73 | pass in something into the constructor in order to do anything |
|---|
| 74 | interesting. An example is offset_separator. This class breaks a |
|---|
| 75 | string into tokens based on offsets for example</p> |
|---|
| 76 | |
|---|
| 77 | <p align="left">12252001 when parsed using offsets of 2,2,4 |
|---|
| 78 | becomes 12 25 2001. Below is an example to parse this.</p> |
|---|
| 79 | <div align="left"> |
|---|
| 80 | |
|---|
| 81 | <pre>// simple_example_3.cpp |
|---|
| 82 | #include<iostream> |
|---|
| 83 | #include<boost/tokenizer.hpp> |
|---|
| 84 | #include<string> |
|---|
| 85 | |
|---|
| 86 | int main(){ |
|---|
| 87 | using namespace std; |
|---|
| 88 | using namespace boost; |
|---|
| 89 | string s = "12252001"; |
|---|
| 90 | int offsets[] = {2,2,4}; |
|---|
| 91 | offset_separator f(offsets, offsets+3); |
|---|
| 92 | tokenizer<offset_separator> tok(s,f); |
|---|
| 93 | for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){ |
|---|
| 94 | cout << *beg << "\n"; |
|---|
| 95 | } |
|---|
| 96 | }</pre> |
|---|
| 97 | </div> |
|---|
| 98 | |
|---|
| 99 | <p align="left"> </p> |
|---|
| 100 | |
|---|
| 101 | <hr> |
|---|
| 102 | |
|---|
| 103 | <p>© Copyright John R. Bandela 2001. Permission to copy, use, |
|---|
| 104 | modify, sell and distribute this document is granted provided |
|---|
| 105 | this copyright notice appears in all copies. This document is |
|---|
| 106 | provided "as is" without express or implied warranty, |
|---|
| 107 | and with no claim as to its suitability for any purpose.</p> |
|---|
| 108 | |
|---|
| 109 | <p align="left"> </p> |
|---|
| 110 | </body> |
|---|
| 111 | </html> |
|---|
