GetStringTokenizer(string, string)

Create a string tokenizer for a given string with a specified delimiter

struct sStringTokenizer GetStringTokenizer(
    string sString,
    string sDelimiter
);

Parameters

sString

The string to be split up into parts (tokens.)

sDelimiter

The delimiter used to split up the string.

Description

This function is to be used in combination with the other functions from x0_i0_stringlib.

It creates and returns a fresh string tokenizer (in form of a sStringTokenizer struct). The tokenizer is required as input for the other functions in x0_i0_stringlib in order to split up a string into parts (tokens).

GetStringTokenizer() requires two arguments: The string that is to be split up into tokens (sString) and the delimiter used to split up the string (sDelimiter). The delimiter MUST be a string containing a single character.

Remarks

Contrary to the description found in x0_i0_stringlib, the DELIMITER MUST be a SINGLE CHARACTER. Otherwise the tokenization process does not work as expected. The reason is that the function AdvanceToNextToken implicitly assumes that the delimiter has a length of 1. It is easy to modify that function in order to accommodate a delimiter consisting of multiple characters, by replacing any occurrence of 'nDelimPos+1' with 'nDelimPos+GetStringLength(stTok.sDelimiter)'. However, such a modification would result in functionality different from the Bioware provided functionality.

Keep in mind that the functions in x0_i0_stringlib are not very efficient. They do a lot of unnecessary string manipulations and unnecessary string parameter passing; string operations and string parameter passing (into or out of nwnscript functions) are amongst the most inefficient operations in nwnscript.

For the purpose of string tokenization by the functions provided in x0_i0_stringlib, a token is any substring (including empty substrings!) within the original string enclosed by the specified delimiter (e.g. every token has one delimiter to the left, one to the right and NO delimiters within). Any non-empty original string is treated as if it were enclosed within a pair of (virtual) delimiters to its left and right. Thus, unless the original string is empty, the number of tokens is always one higher than the number of delimiters contained within the original string. A non-empty string with no delimiters therefore consists out of one token, which is equal to the original string. An empty string has no tokens, although an empty token will be returned on request.

Example:

sString = "I|am|sloppy||programmer";
sDelimiter = "|";

Token[0] = "I";
Token[1] = "am";
Token[2] = "sloppy";
Token[3] = "";
Token[4] = "programmer";

sString contains five tokens and four delimiters.

Known Bugs

Contrary to the description found in the include file, the delimiter must be a single character (see remarks above).

Requirements

#include "x0_i0_stringlib"

Version

???

functions:	AdvanceToNextToken \| FindSubString \| GetNextToken \| GetNumberTokens \| GetSubString \| GetTokenByPosition \| HasMoreTokens
categories:	String Functions