ezEngine  Milestone 7
ezPreprocessor Class Reference

ezPreprocessor implements a standard C preprocessor. It can be used to preprocess files to get the output after macro expansion and #ifdef handling. More...

#include <Preprocessor.h>

Classes

struct  CustomDefine
 
struct  CustomToken
 
struct  FileData
 
struct  IfDefState
 
struct  MacroDefinition
 
struct  ProcessingEvent
 The event data that the processor broadcasts. More...
 

Public Types

enum  IncludeType { MainFile, RelativeInclude, GlobalInclude }
 Describes the type of #include that was encountered during preprocessing. More...
 
typedef ezDelegate< ezResult(const
char *szAbsoluteFile,
ezDynamicArray< ezUInt8 >
&FileContent, ezTimestamp
&out_FileModification)> 
FileOpenCB
 This type of callback is used to read an #include file. szAbsoluteFile is the path that the FileLocatorCB reported, the result needs to be stored in FileContent.
 
typedef ezDelegate< ezResult(const
char *szCurAbsoluteFile, const
char *szIncludeFile,
IncludeType IncType, ezString
&out_sAbsoluteFilePath)> 
FileLocatorCB
 This type of callback is used to retrieve the absolute path of the szIncludeFile when #included inside szCurAbsoluteFile. More...
 
typedef ezDelegate< bool(const
char *szUnknownCommand)> 
PassThroughUnknownCmdCB
 Every time an unknown command (e.g. '#version') is encountered, this callback is used to determine whether the command shall be passed through. More...
 
typedef ezHybridArray< const
ezToken *, 32 > 
TokenStream
 
typedef ezDeque< TokenStreamMacroParameters
 

Public Member Functions

void SetLogInterface (ezLogInterface *pLog)
 All error output is sent to the given ezLogInterface. More...
 
void SetCustomFileCache (ezTokenizedFileCache *pFileCache=nullptr)
 Allows to specify a custom cache object that should be used for storing the tokenized result of files. More...
 
void SetPassThroughPragma (bool bPassThrough)
 If set to true, all #pragma commands are passed through to the output, otherwise they are removed.
 
void SetPassThroughLine (bool bPassThrough)
 If set to true, all #line commands are passed through to the output, otherwise they are removed.
 
void SetPassThroughUnknownCmdsCB (PassThroughUnknownCmdCB callback)
 Sets the callback that is used to determine whether an unknown command is passed through or triggers an error.
 
void SetFileOpenFunction (FileOpenCB OpenAbsFileCB)
 Sets the callback that is needed to read input data. More...
 
void SetFileLocatorFunction (FileLocatorCB LocateAbsFileCB)
 Sets the callback that is needed to locate an input file. More...
 
ezResult AddCustomDefine (const char *szDefinition)
 Adds a #define to the preprocessor, even before any file is processed. More...
 
ezResult Process (const char *szMainFile, TokenStream &TokenOutput)
 Processes the given file and returns the result as a stream of tokens. More...
 
ezResult Process (const char *szMainFile, ezStringBuilder &sOutput, bool bKeepComments=true, bool bRemoveRedundantWhitespace=false, bool bInsertLine=false)
 Processes the given file and returns the result as a string. More...
 

Public Attributes

ezEvent< const ProcessingEvent & > m_ProcessingEvents
 Broadcasts events during the processing. This can be used to create detailed callstacks when an error is encountered. It also broadcasts errors and warnings with more detailed information than the log interface allows.
 

Private Types

enum  IfDefActivity { IsActive, IsInactive, WasActive }
 
enum  TokenFlags : ezUInt32 { NoFurtherExpansion = EZ_BIT(0) }
 

Private Member Functions

ezResult ProcessFile (const char *szFile, TokenStream &TokenOutput)
 
ezResult ProcessCmd (const TokenStream &Tokens, TokenStream &TokenOutput)
 
ezResult OpenFile (const char *szFile, const ezTokenizer **pTokenizer)
 
bool RemoveDefine (const char *szName)
 
ezResult HandleDefine (const TokenStream &Tokens, ezUInt32 &uiCurToken)
 
ezResult StoreDefine (const ezToken *pMacroNameToken, const TokenStream *pReplacementTokens, ezUInt32 uiFirstReplacementToken, ezInt32 iNumParameters, bool bUsesVarArgs)
 
ezResult ExtractParameterName (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezString &sIdentifierName)
 
ezResult EvaluateCondition (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseCondition (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseFactor (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionMul (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionOr (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionAnd (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionPlus (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionShift (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionBitOr (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionBitAnd (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult ParseExpressionBitXor (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezInt64 &iResult)
 
ezResult CopyTokensAndEvaluateDefined (const TokenStream &Source, ezUInt32 uiFirstSourceToken, TokenStream &Destination)
 
bool Accept (const TokenStream &Tokens, ezUInt32 &uiCurToken, const char *szToken, ezUInt32 *pAccepted=nullptr)
 
bool Accept (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezTokenType::Enum Type, ezUInt32 *pAccepted=nullptr)
 
bool Accept (const TokenStream &Tokens, ezUInt32 &uiCurToken, const char *szToken1, const char *szToken2, ezUInt32 *pAccepted=nullptr)
 
bool AcceptUnless (const TokenStream &Tokens, ezUInt32 &uiCurToken, const char *szToken1, const char *szToken2, ezUInt32 *pAccepted=nullptr)
 
ezResult Expect (const TokenStream &Tokens, ezUInt32 &uiCurToken, const char *szToken, ezUInt32 *pAccepted=nullptr)
 
ezResult Expect (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezTokenType::Enum Type, ezUInt32 *pAccepted=nullptr)
 
ezResult Expect (const TokenStream &Tokens, ezUInt32 &uiCurToken, const char *szToken1, const char *szToken2, ezUInt32 *pAccepted=nullptr)
 
ezResult ExpectEndOfLine (const TokenStream &Tokens, ezUInt32 &uiCurToken)
 
void CopyTokensReplaceParams (const TokenStream &Source, ezUInt32 uiFirstSourceToken, TokenStream &Destination, const ezHybridArray< ezString, 16 > &parameters)
 
void CombineTokensToString (const TokenStream &Tokens, ezUInt32 uiCurToken, ezStringBuilder &sResult, bool bKeepComments=true, bool bRemoveRedundantWhitespace=false, bool bInsertLine=false)
 
void CombineRelevantTokensToString (const TokenStream &Tokens, ezUInt32 uiCurToken, ezStringBuilder &sResult)
 
void CreateCleanTokenStream (const TokenStream &Tokens, ezUInt32 uiCurToken, TokenStream &Destination, bool bKeepComments)
 
ezResult Expand (const TokenStream &Tokens, TokenStream &Output)
 
ezResult ExpandOnce (const TokenStream &Tokens, TokenStream &Output)
 
ezResult ExpandObjectMacro (MacroDefinition &Macro, TokenStream &Output, const ezToken *pMacroToken)
 
ezResult ExpandFunctionMacro (MacroDefinition &Macro, const MacroParameters &Parameters, TokenStream &Output, const ezToken *pMacroToken)
 
ezResult ExpandMacroParam (const ezToken &MacroToken, ezUInt32 uiParam, TokenStream &Output, const MacroDefinition &Macro)
 
void PassThroughFunctionMacro (MacroDefinition &Macro, const MacroParameters &Parameters, TokenStream &Output)
 
ezTokenAddCustomToken (const ezToken *pPrevious, const char *szNewText)
 
void OutputNotExpandableMacro (MacroDefinition &Macro, TokenStream &Output)
 
ezResult ExtractAllMacroParameters (const TokenStream &Tokens, ezUInt32 &uiCurToken, ezDeque< TokenStream > &AllParameters)
 
ezResult ExtractParameterValue (const TokenStream &Tokens, ezUInt32 &uiCurToken, TokenStream &ParamTokens)
 
ezResult InsertParameters (const TokenStream &Tokens, TokenStream &Output, const MacroDefinition &Macro)
 
ezResult InsertStringifiedParameters (const TokenStream &Tokens, TokenStream &Output, const MacroDefinition &Macro)
 
ezResult ConcatenateParameters (const TokenStream &Tokens, TokenStream &Output, const MacroDefinition &Macro)
 
void MergeTokens (const ezToken *pFirst, const ezToken *pSecond, TokenStream &Output, const MacroDefinition &Macro)
 
ezTokenCreateStringifiedParameter (ezUInt32 uiParam, const ezToken *pParamToken, const MacroDefinition &Macro)
 
ezResult HandleErrorDirective (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken)
 
ezResult HandleWarningDirective (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken)
 
ezResult HandleUndef (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken)
 
ezResult HandleEndif (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken)
 
ezResult HandleElif (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken)
 
ezResult HandleIf (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken)
 
ezResult HandleElse (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken)
 
ezResult HandleIfdef (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken, bool bIsIfdef)
 
ezResult HandleInclude (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken, TokenStream &TokenOutput)
 
ezResult HandleLine (const TokenStream &Tokens, ezUInt32 uiCurToken, ezUInt32 uiDirectiveToken, TokenStream &TokenOutput)
 

Static Private Member Functions

static ezResult DefaultFileLocator (const char *szCurAbsoluteFile, const char *szIncludeFile, ezPreprocessor::IncludeType IncType, ezString &out_sAbsoluteFilePath)
 
static ezResult DefaultFileOpen (const char *szAbsoluteFile, ezDynamicArray< ezUInt8 > &FileContent, ezTimestamp &out_FileModification)
 
static void SkipWhitespace (const TokenStream &Tokens, ezUInt32 &uiCurToken)
 
static void SkipWhitespaceAndNewline (const TokenStream &Tokens, ezUInt32 &uiCurToken)
 
static bool IsEndOfLine (const TokenStream &Tokens, ezUInt32 uiCurToken, bool bIgnoreWhitespace)
 
static void CopyRelevantTokens (const TokenStream &Source, ezUInt32 uiFirstSourceToken, TokenStream &Destination, bool bPreserveNewLines)
 
static void StringifyTokens (const TokenStream &Tokens, ezStringBuilder &sResult, bool bSurroundWithQuotes)
 

Private Attributes

bool m_bPassThroughPragma
 
bool m_bPassThroughLine
 
PassThroughUnknownCmdCB m_PassThroughUnknownCmdCB
 
ezTokenizedFileCache m_InternalFileCache
 
ezTokenizedFileCachem_pUsedFileCache
 
ezDeque< FileDatam_sCurrentFileStack
 
ezLogInterfacem_pLog
 
ezDeque< CustomDefinem_CustomDefines
 
ezDeque< IfDefStatem_IfdefActiveStack
 
FileOpenCB m_FileOpenCallback
 
FileLocatorCB m_FileLocatorCallback
 
ezSet< ezStringm_PragmaOnce
 
ezMap< ezString, MacroDefinitionm_Macros
 
ezToken m_ParameterTokens [32]
 
ezToken m_TokenFile
 
ezToken m_TokenLine
 
const ezTokenm_TokenOpenParenthesis
 
const ezTokenm_TokenClosedParenthesis
 
const ezTokenm_TokenComma
 
ezDeque< const MacroParameters * > m_MacroParamStack
 
ezDeque< const MacroParameters * > m_MacroParamStackExpanded
 
ezDeque< CustomTokenm_CustomTokens
 

Static Private Attributes

static const ezInt32 s_MacroParameter0 = ezTokenType::ENUM_COUNT + 2
 
static ezString s_ParamNames [32]
 

Detailed Description

ezPreprocessor implements a standard C preprocessor. It can be used to preprocess files to get the output after macro expansion and #ifdef handling.

For a detailed documentation about the C preprocessor, see https://gcc.gnu.org/onlinedocs/cpp/

This class implements all standard features:

  • object and function macros
  • Full evaluation of #if, #ifdef etc. including mathematical operations such as #if A > 42
  • Parameter stringification
  • Parameter concatenation
  • LINE and FILE macros
  • Fully correct #line evaluation for error output
  • Correct handling of VA_ARGS
  • #include handling
  • #pragma once
  • #warning and #error for custom failure messages

Class Documentation

struct ezPreprocessor::CustomDefine
Class Members
ezDynamicArray< ezUInt8 > m_Content
ezTokenizer m_Tokenized
struct ezPreprocessor::CustomToken
Class Members
ezString m_sIdentifierString
ezToken m_Token

Member Typedef Documentation

typedef ezDelegate<ezResult (const char* szCurAbsoluteFile, const char* szIncludeFile, IncludeType IncType, ezString& out_sAbsoluteFilePath)> ezPreprocessor::FileLocatorCB

This type of callback is used to retrieve the absolute path of the szIncludeFile when #included inside szCurAbsoluteFile.

Note that you should ensure that out_sAbsoluteFilePath is always identical (including casing and path slashes) when it is supposed to point to the same file, as this exact name is used for file lookup (and therefore also file caching). If it is not identical, file caching will not work, and on different OSes the file may be found or not.

typedef ezDelegate<bool (const char* szUnknownCommand)> ezPreprocessor::PassThroughUnknownCmdCB

Every time an unknown command (e.g. '#version') is encountered, this callback is used to determine whether the command shall be passed through.

If the callback returns false, an error is generated and parsing fails. The callback thus acts as a whitelist for all commands that shall be passed through.

Member Enumeration Documentation

Describes the type of #include that was encountered during preprocessing.

Enumerator
MainFile 

This is used for the very first access to the main source file.

RelativeInclude 

An #include "file" has been encountered.

GlobalInclude 

An #include <file> has been encountered.

Member Function Documentation

ezResult ezPreprocessor::AddCustomDefine ( const char *  szDefinition)

Adds a #define to the preprocessor, even before any file is processed.

This allows to have global macros that are always defined for all processed files, such as the current platform etc. szDefinition must be in the form of the test that follows a #define statement. So to define the macro "WIN32", just pass that string. You can define any macro that could also be defined in the source files.

If the definition is invalid, EZ_FAILURE is returned. Also the preprocessor might end up in an invalid state, so using it any further might fail (including crashing).

ezResult ezPreprocessor::HandleDefine ( const TokenStream Tokens,
ezUInt32 &  uiCurToken 
)
private
Todo:
Make sure the same parameter name is not used twice
void ezPreprocessor::PassThroughFunctionMacro ( MacroDefinition Macro,
const MacroParameters Parameters,
TokenStream Output 
)
private
Todo:
Maybe the passed through parameters need expansion
ezResult ezPreprocessor::Process ( const char *  szMainFile,
TokenStream TokenOutput 
)

Processes the given file and returns the result as a stream of tokens.

This function is useful when you want to further process the output afterwards and thus need it in a tokenized form anyway.

ezResult ezPreprocessor::Process ( const char *  szMainFile,
ezStringBuilder sOutput,
bool  bKeepComments = true,
bool  bRemoveRedundantWhitespace = false,
bool  bInsertLine = false 
)

Processes the given file and returns the result as a string.

This function creates a string from the tokenized result. If bKeepComments is true, all block and line comments are included in the output string, otherwise they are removed.

void ezPreprocessor::SetCustomFileCache ( ezTokenizedFileCache pFileCache = nullptr)

Allows to specify a custom cache object that should be used for storing the tokenized result of files.

This allows to share one cache across multiple instances of ezPreprocessor and across time. E.g. it makes it possible to prevent having to read and tokenize include files that are referenced often.

void ezPreprocessor::SetFileLocatorFunction ( FileLocatorCB  LocateAbsFileCB)

Sets the callback that is needed to locate an input file.

The default file locator will assume that the main source file and all files #included in angle brackets can be opened without modification. Files #included in "" will be appended as relative paths to the path of the file they appeared in.

void ezPreprocessor::SetFileOpenFunction ( FileOpenCB  OpenAbsFileCB)

Sets the callback that is needed to read input data.

The default file open function will just try to open files via ezFileReader.

void ezPreprocessor::SetLogInterface ( ezLogInterface pLog)

All error output is sent to the given ezLogInterface.

Note that when the preprocessor encounters any error, it will stop immediately and usually no output is generated. However, there are also a few cases where only a warning is generated, in this case preprocessing will continue without problems.

Additionally errors and warnings are also broadcast through m_ProcessingEvents. So if you want to output more detailed information, that method should be preferred, because the events carry more information about the current file and line number etc.


The documentation for this class was generated from the following files: