CPreprocessor::Token Class Reference

A input token. More...

Public Types
enum	Kind {   TK_EOS, TK_ERROR, TK_WHITESPACE, TK_NEWLINE,   TK_LINECONT, TK_NUMBER, TK_KEYWORD, TK_PUNCTUATION,   TK_DIRECTIVE, TK_STRING, TK_COMMENT, TK_LINECOMMENT,   TK_TEXT }

Public Member Functions
	Token ()
	Token (Kind iType)
	Token (Kind iType, const char *iString, size_t iLength)
	Token (const Token &iOther)
	~Token ()
Token &	operator= (const Token &iOther)
	Assignment operator. More...
void	Append (const char *iString, size_t iLength)
	Append a string to this token. More...
void	Append (const Token &iOther)
	Append a token to this token. More...
void	AppendNL (int iCount)
	Append given number of newlines to this token. More...
int	CountNL ()
	Count number of newlines in this token. More...
bool	GetValue (long &oValue) const
	Get the numeric value of the token. More...
void	SetValue (long iValue)
	Set the numeric value of the token. More...
bool	operator== (const Token &iOther)
	Test two tokens for equality. More...

Public Attributes
Kind	Type
	Token type. More...
size_t	Allocated
	True if string was allocated (and must be freed) More...
union {
const char *   String
	A pointer somewhere into the input buffer. More...
char *   Buffer
	A memory-allocated string. More...
};
size_t	Length
	Token length in bytes. More...

Detailed Description

A input token.

For performance reasons most tokens will point to portions of the input stream, so no unneeded memory allocation is done. However, in some cases we must allocate different memory for token storage, in this case this is signalled by setting the Allocated member to non-zero in which case the destructor will know that it must free memory on object destruction.

Again for performance reasons we use malloc/realloc/free here because C++-style new[] lacks the realloc() counterpart.

Member Enumeration Documentation

enum CPreprocessor::Token::Kind

Enumerator
TK_EOS
TK_ERROR
TK_WHITESPACE
TK_NEWLINE
TK_LINECONT
TK_NUMBER
TK_KEYWORD
TK_PUNCTUATION
TK_DIRECTIVE
TK_STRING
TK_COMMENT
TK_LINECOMMENT
TK_TEXT

Constructor & Destructor Documentation

CPreprocessor::Token::Token ( )

inline

CPreprocessor::Token::Token ( Kind  iType )

inline

CPreprocessor::Token::Token ( Kind  iType, const char *  iString, size_t  iLength  )

inline

CPreprocessor::Token::Token ( const Token &  iOther )

inline

CPreprocessor::Token::~Token ( )

inline

Member Function Documentation

void CPreprocessor::Token::Append	(	const char *	iString,
		size_t	iLength
	)

Append a string to this token.

void CPreprocessor::Token::Append

(

const Token &

iOther

)

Append a token to this token.

void CPreprocessor::Token::AppendNL

(

int

iCount

)

Append given number of newlines to this token.

int CPreprocessor::Token::CountNL

(

)

Count number of newlines in this token.

bool CPreprocessor::Token::GetValue

(

long &

oValue

)

const

Get the numeric value of the token.

Token& CPreprocessor::Token::operator= ( const Token &  iOther )

inline

Assignment operator.

bool CPreprocessor::Token::operator== ( const Token &  iOther )

inline

Test two tokens for equality.

void CPreprocessor::Token::SetValue

(

long

iValue

)

Set the numeric value of the token.

Member Data Documentation

union { ... }

size_t CPreprocessor::Token::Allocated

mutable

True if string was allocated (and must be freed)

char* CPreprocessor::Token::Buffer

A memory-allocated string.

size_t CPreprocessor::Token::Length

Token length in bytes.

const char* CPreprocessor::Token::String

A pointer somewhere into the input buffer.

Kind CPreprocessor::Token::Type

Token type.

---

The documentation for this class was generated from the following files:

• /var/svn/checkout/source/ps/Preprocessor.h
• /var/svn/checkout/source/ps/Preprocessor.cpp