| 1 | <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
|---|
| 2 | |
|---|
| 3 | <html> |
|---|
| 4 | <head> |
|---|
| 5 | <meta http-equiv="Content-Language" content="en-us"> |
|---|
| 6 | <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> |
|---|
| 7 | <meta name="GENERATOR" content="Microsoft FrontPage 6.0"> |
|---|
| 8 | <meta name="ProgId" content="FrontPage.Editor.Document"> |
|---|
| 9 | |
|---|
| 10 | <title>Introduction</title> |
|---|
| 11 | </head> |
|---|
| 12 | |
|---|
| 13 | <body bgcolor="#FFFFFF"> |
|---|
| 14 | <p><img src="../../boost.png" alt="C++ Boost" width="277" height= |
|---|
| 15 | "86"><br></p> |
|---|
| 16 | |
|---|
| 17 | <h1 align="center">Introduction</h1> |
|---|
| 18 | |
|---|
| 19 | <p align="left">The boost Tokenizer package provides a flexible and easy to |
|---|
| 20 | use way to break of a string or other character sequence into a series of |
|---|
| 21 | tokens. Below is a simple example that will break up a phrase into |
|---|
| 22 | words.</p> |
|---|
| 23 | |
|---|
| 24 | <div align="left"> |
|---|
| 25 | <pre> |
|---|
| 26 | // simple_example_1.cpp |
|---|
| 27 | #include<iostream> |
|---|
| 28 | #include<boost/tokenizer.hpp> |
|---|
| 29 | #include<string> |
|---|
| 30 | |
|---|
| 31 | int main(){ |
|---|
| 32 | using namespace std; |
|---|
| 33 | using namespace boost; |
|---|
| 34 | string s = "This is, a test"; |
|---|
| 35 | tokenizer<> tok(s); |
|---|
| 36 | for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){ |
|---|
| 37 | cout << *beg << "\n"; |
|---|
| 38 | } |
|---|
| 39 | } |
|---|
| 40 | </pre> |
|---|
| 41 | </div> |
|---|
| 42 | |
|---|
| 43 | <p align="left">You can choose how the string gets broken up. You do this |
|---|
| 44 | by specifying the TokenizerFunction. If you do not specify anything, the |
|---|
| 45 | default TokenizerFunction is char_delimiters_separator<char> which |
|---|
| 46 | defaults to breaking up a string based on space and punctuation. Here is an |
|---|
| 47 | example of using another TokenizerFunction called escaped_list_separator. |
|---|
| 48 | This TokenizerFunction parses a superset of comma separated value (csv) |
|---|
| 49 | lines. The format looks like this</p> |
|---|
| 50 | |
|---|
| 51 | <p align="left">Field 1,"putting quotes around fields, allows commas",Field |
|---|
| 52 | 3</p> |
|---|
| 53 | |
|---|
| 54 | <p align="left">Below is an example that will break the previous line into |
|---|
| 55 | its 3 fields</p> |
|---|
| 56 | |
|---|
| 57 | <div align="left"> |
|---|
| 58 | <pre> |
|---|
| 59 | // simple_example_2.cpp |
|---|
| 60 | #include<iostream> |
|---|
| 61 | #include<boost/tokenizer.hpp> |
|---|
| 62 | #include<string> |
|---|
| 63 | |
|---|
| 64 | int main(){ |
|---|
| 65 | using namespace std; |
|---|
| 66 | using namespace boost; |
|---|
| 67 | string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3"; |
|---|
| 68 | tokenizer<escaped_list_separator<char> > tok(s); |
|---|
| 69 | for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){ |
|---|
| 70 | cout << *beg << "\n"; |
|---|
| 71 | } |
|---|
| 72 | } |
|---|
| 73 | </pre> |
|---|
| 74 | </div> |
|---|
| 75 | |
|---|
| 76 | <p align="left">Finally, for some TokenizerFunctions you have to pass in |
|---|
| 77 | something into the constructor in order to do anything interesting. An |
|---|
| 78 | example is offset_separator. This class breaks a string into tokens based |
|---|
| 79 | on offsets for example</p> |
|---|
| 80 | |
|---|
| 81 | <p align="left">12252001 when parsed using offsets of 2,2,4 becomes 12 25 |
|---|
| 82 | 2001. Below is an example to parse this.</p> |
|---|
| 83 | |
|---|
| 84 | <div align="left"> |
|---|
| 85 | <pre> |
|---|
| 86 | // simple_example_3.cpp |
|---|
| 87 | #include<iostream> |
|---|
| 88 | #include<boost/tokenizer.hpp> |
|---|
| 89 | #include<string> |
|---|
| 90 | |
|---|
| 91 | int main(){ |
|---|
| 92 | using namespace std; |
|---|
| 93 | using namespace boost; |
|---|
| 94 | string s = "12252001"; |
|---|
| 95 | int offsets[] = {2,2,4}; |
|---|
| 96 | offset_separator f(offsets, offsets+3); |
|---|
| 97 | tokenizer<offset_separator> tok(s,f); |
|---|
| 98 | for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){ |
|---|
| 99 | cout << *beg << "\n"; |
|---|
| 100 | } |
|---|
| 101 | } |
|---|
| 102 | </pre> |
|---|
| 103 | </div> |
|---|
| 104 | |
|---|
| 105 | <p align="left"> </p> |
|---|
| 106 | <hr> |
|---|
| 107 | |
|---|
| 108 | <p><a href="http://validator.w3.org/check?uri=referer"><img border="0" src= |
|---|
| 109 | "http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01 Transitional" |
|---|
| 110 | height="31" width="88"></a></p> |
|---|
| 111 | |
|---|
| 112 | <p>Revised |
|---|
| 113 | <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p> |
|---|
| 114 | |
|---|
| 115 | <p><i>Copyright © 2001 John R. Bandela</i></p> |
|---|
| 116 | |
|---|
| 117 | <p><i>Distributed under the Boost Software License, Version 1.0. (See |
|---|
| 118 | accompanying file <a href="../../LICENSE_1_0.txt">LICENSE_1_0.txt</a> or |
|---|
| 119 | copy at <a href= |
|---|
| 120 | "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</a>)</i></p> |
|---|
| 121 | </body> |
|---|
| 122 | </html> |
|---|
