Add a User-defined Token Function

In GSQL’s Data Definition and Loading (DDL) language, users can define their own token functions if the built-in token functions do not meet their needs. This guide describes the procedures to define token functions.

Procedure

Step 1: Download the current TokenBank.cpp file

All user-defined token functions are kept in a file named tokenbank.cpp. Use the GET TokenBank command in GSQL to download the current UDF file to any location on your machine. The path after the keyword TO specifies the path where the file will be output to.

The file and the directories will be created if they do not exist, and the file must end with the file extension .hpp. An example is shown below:

GSQL > GET TokenBank TO "/home/tigergraph/TokenBank.cpp"
GET TokenBank successfully.

If you only supply a directory but not a filename, the file will be created with the default filename TokenBank.cpp.

Step 2: Define token function in C++

Define the token function in the file you just downloaded. The function should be a C++ function. The function can either return a value that is used for an attribute expression or used in a WHERE clause as a condition expression. Depending on the return type of the function, the signature of the function must match the allowed format.

If your token function is used to return an attribute expression, the signature of the function must follow the format specified in the table below depending on the attribute type.

Attribute type Function signature Function return type

STRING or STRING COMPRESS

extern "C" void funcName (const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum, char* const oToken, uint32_t& oTokenLen)

void. The value of oToken will be returned in GSQL

BOOL

+extern "C" bool funcName (const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum) +

bool

UINT

+extern "C" uint64_t funcName (const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum) +

uint64_t

INT

+extern "C" int64_t funcName (const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum) +

int64_t

FLOAT

+extern "C" float funcName (const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum) +

float

DOUBLE

+extern "C" double funcName (const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum) +

double

The parameters iToken, iTokenLen, and iTokenNum must be named exactly as such, and are used to describe the input tokens:

  • iToken is an array of the string tokens,

  • iTokenLen is an array of the lengths of the string tokens

  • iTokenNum is the number of tokens.

For token functions for attribute types STRING or STRING COMPRESS, the return type for the C++ function is void. Use the parameter oToken to store the string you want returned, and in GSQL the token function will return the string stored in oToken:

  • oToken is the returned string value

  • oTokenLen is the length of the return string

Note that the input tokens are always in string (char*) format. If necessary, convert them to other types inside the function.

Example

The built-in token function gsql_concat is used as an example below. It takes multiple token parameters and returns a string.

extern "C" void gsql_concat(const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum, char* const oToken, uint32_t& oTokenLen) {
  int k = 0;
  for (int i=0; i < iTokenNum; i++) {
    for (int j =0; j < iTokenLen[i]; j++) {
           oToken[k++] = iToken[i][j];
    }
  }
  oTokenLen = k;
}

User-defined Token Functions for WHERE Clause

User-defined token functions (described above) can also be used to construct the boolean conditional expression in the WHERE clause. However, there are some restrictions in the WHERE clause:

In the clause "`WHERE \'\'*conditions*` ",

  • The only user-defined token functions allowed are those that return a boolean value.

  • If a user-defined token function is used in a WHERE clause, then it must constitute the entire condition; it cannot be combined with another function or operator to produce a subsequent value. However, the arguments of the UDF can include other functions.

The source code for the built-in token function gsql_token_equal is used as an example for how to write a user-defined token function.

extern "C" bool gsql_token_equal(const char* const iToken[], uint32_t iTokenLen[], uint32_t iTokenNum) {
  if (iTokenNum != 2) {
    return false;
  }
  if (iTokenLen[0] != iTokenLen[1]) {
    return false;
  }
  for (int i =0; i < iTokenLen[0]; i++) {
    if (iToken[0][i] != iToken[1][i]) {
      return false;
    }
  }
  return true;
}

Step 3: Upload the TokenBank.cpp file

After defining the token functions, use the PUT TokenBank command to upload the functions. The path after the keyword FROM is the absolute path to the TokenBank.cpp file. An example is shown below:

GSQL > PUT TokenBank FROM "/home/tigergraph/TokenBank.cpp"
PUT TokenBank successfully.