Lines Matching refs:tokenizer
90786 ** directly to the parser from the tokenizer.
93839 assert( TOKEN.z[0] ); /* The tokenizer always gives us a token */
94046 ** An tokenizer for SQL
94847 ** An tokenizer for SQL
94850 ** This code used to be part of the tokenizer.c source file. But by
97946 ** generated by the tokenizer, while an "offset" is a byte offset,
98179 ** sqlite3_tokenizer_module is a singleton defining the tokenizer
98183 ** sqlite3_tokenizer is used to define a particular tokenizer, perhaps
98186 ** sqlite3_tokenizer_cursor is generated by a tokenizer to generate
98198 ** Structures used by the tokenizer interface. When a new tokenizer
98204 ** the tokenizer clause of the CREATE VIRTUAL TABLE statement to the
98205 ** sqlite3_tokenizer_module.xCreate() function of the requested tokenizer
98207 ** sqlite3_tokenizer structure representing the specific tokenizer to
98208 ** be used for the fts3 table (customized by the tokenizer clause arguments).
98228 ** Create a new tokenizer. The values in the argv[] array are the
98229 ** arguments passed to the "tokenizer" clause of the CREATE VIRTUAL
98233 ** CREATE .. USING fts3( ... , tokenizer <tokenizer-name> arg1 arg2)
98240 ** to point at the newly created tokenizer structure. The generic
98247 sqlite3_tokenizer **ppTokenizer /* OUT: Created tokenizer */
98251 ** Destroy an existing tokenizer. The fts3 module calls this method
98257 ** Create a tokenizer cursor to tokenize an input buffer. The caller
98264 sqlite3_tokenizer_cursor **ppCursor /* OUT: Created tokenizer cursor */
98268 ** Destroy an existing tokenizer cursor. The fts3 module calls this
98274 ** Retrieve the next token from the tokenizer cursor pCursor. This
98289 ** The buffer *ppToken is set to point at is managed by the tokenizer
98307 const sqlite3_tokenizer_module *pModule; /* The module for this tokenizer */
98518 sqlite3_tokenizer *pTokenizer; /* tokenizer for inserts and queries */
98864 /* Invoke the tokenizer destructor to free the tokenizer. */
99017 const char *zTokenizer = 0; /* Name of tokenizer to use */
99128 void *pAux, /* Pointer to tokenizer hash table */
99138 void *pAux, /* Pointer to tokenizer hash table */
100796 ** allocated for the tokenizer hash table.
100811 ** to by the argument to point a the "simple" tokenizer implementation.
100813 ** porter tokenizer/stemmer implementation.
100920 ** syntax is relatively simple, the whole tokenizer/parser system is
101012 ** Extract the next token from buffer z (length n) using the tokenizer
101585 ** The first parameter, pTokenizer, is passed the fts3 tokenizer module to
101728 ** fts3_exprtest(<tokenizer>, <expr>, <column 1>, ...);
101730 ** The first argument, <tokenizer>, is the name of the fts3 tokenizer used
101756 "Usage: fts3_exprtest(tokenizer, expr, col1, ...", -1
101767 sqlite3_result_error(context, "No such tokenizer module", -1);
102220 ** Implementation of the full-text-search tokenizer that implements
102260 ** Create a new tokenizer instance.
102279 ** Destroy a tokenizer
102293 sqlite3_tokenizer *pTokenizer, /* The tokenizer */
102825 ** The set of routines that implement the porter-stemmer tokenizer
102837 ** Allocate a new porter tokenizer. Return a pointer to the new
102838 ** tokenizer in *ppModule
102863 ** This particular file implements the generic tokenizer interface.
102935 char *zErr = sqlite3_mprintf("unknown tokenizer: %s", zName);
102998 const char *zArg, /* Possible tokenizer specification */
103000 const char **pzTokenizer, /* OUT: Set to zArg if is tokenizer */
103031 *pzErr = sqlite3_mprintf("unknown tokenizer: %s", z);
103054 *pzErr = sqlite3_mprintf("unknown tokenizer");
103086 ** using the built-in "simple" tokenizer:
103137 char *zErr = sqlite3_mprintf("unknown tokenizer: %s", zName);
103241 ** in the README.tokenizer file as an example, so it is important to
103272 assert( 0==strcmp(sqlite3_errmsg(db), "unknown tokenizer: nosuchtokenizer") );
103357 ** Implementation of the "simple" full-text-search tokenizer.
103395 ** Create a new tokenizer instance.
103436 ** Destroy a tokenizer
103450 sqlite3_tokenizer *pTokenizer, /* The tokenizer */
103546 ** The set of routines that implement the simple tokenizer
103558 ** Allocate a new simple tokenizer. Return a pointer to the new
103559 ** tokenizer in *ppModule
105943 ** tokenizer.
106140 const sqlite3_tokenizer_module *pTModule; /* The tokenizer module */
106141 sqlite3_tokenizer *pTokenizer; /* The specific tokenizer */
106148 const char *zToken; /* Next token from the tokenizer */
106972 const char *ZDUMMY; /* Dummy arguments used with tokenizer */
106973 int DUMMY1, DUMMY2, DUMMY3; /* Dummy arguments used with tokenizer */
110615 ** This file implements a tokenizer for fts3 based on the ICU library.
110650 ** Create a new tokenizer instance.
110655 sqlite3_tokenizer **ppTokenizer /* OUT: Created tokenizer */
110680 ** Destroy a tokenizer
110695 sqlite3_tokenizer *pTokenizer, /* The tokenizer */
110837 ** The set of routines that implement the simple tokenizer
110849 ** Set *ppModule to point at the implementation of the ICU tokenizer.