Chapter 22. Extending MySQL
options] [test_name]
In the mysql-test directory, you can run an
individual test case with ./mysql-test-run.pl
test_name.
If you have a question about the test suite, or have a test case
to contribute, send an email message to the MySQL
internals mailing list. See
Section 1.5.1, “MySQL Mailing Lists”. This list does not accept
attachments, so you should FTP all the relevant files to:
ftp://ftp.mysql.com/pub/mysql/upload/
MySQL 5.1 and up supports a plugin API that allows the loading and unloading of server components at runtime, without restarting the server. The components supported by this operation include, but are not limited to, full-text parser plugins, storage-engines and server extensions. In MySQL 5.1, the storage engines can be included in the server as oplugins.
Full-text parser plugins can be used to replace or augment the built-in full-text parser. For example, a plugin can parse text into words using rules that differ from those used by the built-in parser. This can be useful if you need to parse text with characteristics different from those expected by the built-in parser.
The plugin interface is intended as the successor to the older user-defined function (UDF) interface. The plugin interface eventually will include an API for creating UDFs, and it is intended this plugin UDF API will replace the older nonplugin UDF API. After that point, it will be possible for UDFs to be revised for use as plugin UDFs so that they can take advantage of the better security and versioning capabilities of the plugin API. Eventually, support for the older UDF API will be phased out.
The plugin interface requires the plugin table
in the mysql database. This table is created as
part of the MySQL installation process. If you are upgrading from
a version older than MySQL 5.1, you should run the
mysql_upgrade command to create this table. See
Section 4.4.8, “mysql_upgrade — Check Tables for MySQL Upgrade”.
For more information on the Plugin API and how it can be used with storage engines, see MySQL Internals: Custom Engine.
In some respects, the plugin API is similar to the older user-defined function (UDF) API that it supersedes, but the plugin API has several advantages over the older interface:
The plugin framework is extendable to accommodate different kinds of plugins.
Some aspects of the plugin API are common to all types of plugins, but the API also allows for type-specific interface elements so that different types of plugins can be created. A plugin with one purpose can have an interface most appropriate to its own requirements and not the requirements of some other plugin type.
Although only the interface for full-text parser plugins is implemented currently, others can be added, such as an interface for UDF plugins.
The plugin API includes versioning information.
The version information included in the plugin API enables a plugin library and each plugin that it contains to be self-identifying with respect to the API version that was used to build the library. If the API changes over time, the version numbers will change, but a server can examine a given plugin library's version information to determine whether it supports the plugins in the library.
There are two types of version numbers. The first is the version for the general plugin framework itself. Each plugin library includes this kind of version number. The second type of version applies to individual plugins. Each specific type of plugin has a version for its interface, so each plugin in a library has a type-specific version number. For example, library containing a full-text parsing plugin has a general plugin API version number, and the plugin has a version number specific to the full-text plugin interface.
Plugin security is improved relative to the UDF interface.
The older interface for writing nonplugin UDFs allowed libraries to be loaded from any directory searched by the system's dynamic linker, and the symbols that identified the UDF library were relatively nonspecific. The newer rules are more strict. A plugin library must be installed in a specific dedicated directory for which the location is controlled by the server and cannot be changed at runtime. Also, the library must contain specific symbols that identify it as a plugin library. The server will not load something as a plugin if it was not built as a plugin.
The newer plugin interface eliminates the security issues of the older UDF interface. When a UDF plugin type is implemented, that will allow nonplugin UDFs to be brought into the plugin framework and the older interface to be phased out.
The plugin implementation includes the following components:
Source files (the locations given indicate where the files are found in a MySQL source distribution):
include/mysql/plugin.hexposes the public plugin API. This file should be examined by anyone who wants to write a plugin library.sql/sql_plugin.handsql/sql_plugin.cccomprise the internal plugin implementation. These files need not be consulted by plugin writers. They may be of interest for those who want to know more about how the server handles plugins.
System table:
The
plugintable in themysqldatabase lists each installed plugin and is required for plugin use. For new MySQL installations, this table is created during the installation process. If you are upgrading from a version older than MySQL 5.1, you should run mysql_upgrade to update your system tables and create theplugintable (see Section 4.4.8, “mysql_upgrade — Check Tables for MySQL Upgrade”).
SQL statements:
INSTALL PLUGINregisters a plugin in theplugintable and loads the plugin code.UNINSTALL PLUGINunregisters a plugin from theplugintable and unloads the plugin code.The
WITH PARSERclause for full-text index creation associates a full-text parser plugin with a givenFULLTEXTindex.SHOW PLUGINSdisplays information about known plugins. ThePLUGINStable inINFORMATION_SCHEMAalso contains plugin information.
System variable:
plugin_dirindicates the location of the directory where all plugins must be installed. The value of this variable can be specified at server startup with a--plugin_dir=option.path
MySQL has a built-in parser that it uses by default for full-text operations (parsing text to be indexed, or parsing a query string to determine the terms to be used for a search). For full-text processing, “parsing” means extracting words from text or a query string based on rules that define which character sequences make up a word and where word boundaries lie.
When parsing for indexing purposes, the parser passes each word to the server, which adds it to a full-text index. When parsing a query string, the parser passes each word to the server, which accumulates the words for use in a search.
The parsing properties of the built-in full-text parser are
described in Section 11.8, “Full-Text Search Functions”. These properties
include rules for determining how to extract words from text.
The parser is influenced by certain system variables such as
ft_min_word_len and
ft_max_word_len that cause
words shorter or longer to be excluded, and by the stopword list
that identifies common words to be ignored.
The plugin API enables you to provide a full-text parser of your own so that you have control over the basic duties of a parser. A parser plugin can operate in either of two roles:
The plugin can replace the built-in parser. In this role, the plugin reads the input to be parsed, splits it up into words, and passes the words to the server (either for indexing or for word accumulation).
One reason to use a parser this way is that you need to use different rules from those of the built-in parser for determining how to split up input into words. For example, the built-in parser considers the text “case-sensitive” to consist of two words “case” and “sensitive,” whereas an application might need to treat the text as a single word.
The plugin can act in conjunction with the built-in parser by serving as a front end for it. In this role, the plugin extracts text from the input and passes the text to the parser, which splits up the text into words using its normal parsing rules. In particular, this parsing will be affected by the
ft_system variables and the stopword list.xxxOne reason to use a parser this way is that you need to index content such as PDF documents, XML documents, or
.docfiles. The built-in parser is not intended for those types of input but a plugin can pull out the text from these input sources and pass it to the built-in parser.
It is also possible for a parser plugin to operate in both roles. That is, it could extract text from nonplaintext input (the front end role), and also parse the text into words (thus replacing the built-in parser).
A full-text plugin is associated with full-text indexes on a
per-index basis. That is, when you install a parser plugin
initially, that does not cause it to be used for any full-text
operations. It simply becomes available. For example, a
full-text parser plugin becomes available to be named in a
WITH PARSER clause when creating individual
FULLTEXT indexes. To create such an index at
table-creation time, do this:
CREATE TABLE t ( doc CHAR(255), FULLTEXT INDEX (doc) WITH PARSER my_parser );
Or you can add the index after the table has been created:
ALTER TABLE t ADD FULLTEXT INDEX (doc) WITH PARSER my_parser;
The only SQL change for associating the parser with the index is
the WITH PARSER clause. Searches are
specified as before, with no changes needed for queries.
When you associate a parser plugin with a
FULLTEXT index, the plugin is required for
using the index. If the parser plugin is dropped, any index
associated with it becomes unusable. Any attempt to use it a
table for which a plugin is not available results in an error,
although DROP TABLE is still
possible.
This section describes the general and type-specific parts of
the plugin API. It also provides a step-by-step guide to
creating a plugin library. For example plugin source code, see
the plugin/fulltext directory of a MySQL
source distribution.
You can write plugins in C or C++ (or another language that can use C calling conventions). Plugins are loaded and unloaded dynamically, so your operating system must support dynamic loading and you must have compiled mysqld dynamically (not statically).
A plugin contains code that becomes part of the running server,
so when you write a plugin, you are bound by any and all
constraints that otherwise apply to writing server code. For
example, you may have problems if you attempt to use functions
from the libstdc++ library. These constraints
may change in future versions of the server, so it is possible
that server upgrades will require revisions to plugins that were
originally written for older servers. For information about
these constraints, see Section 2.10.2, “Typical configure Options”, and
Section 2.10.4, “Dealing with Problems Compiling MySQL”.
Every plugin must have a general plugin declaration. The
declaration corresponds to the
st_mysql_plugin structure in the
plugin.h file:
struct st_mysql_plugin
{
int type; /* the plugin type (a MYSQL_XXX_PLUGIN value) */
void *info; /* pointer to type-specific plugin descriptor */
const char *name; /* plugin name */
const char *author; /* plugin author (for SHOW PLUGINS) */
const char *descr; /* general descriptive text (for SHOW PLUGINS ) */
int license; /* the plugin license (PLUGIN_LICENSE_XXX) */
int (*init)(void *); /* the function to invoke when plugin is loaded */
int (*deinit)(void *);/* the function to invoke when plugin is unloaded */
unsigned int version; /* plugin version (for SHOW PLUGINS) */
struct st_mysql_show_var *status_vars;
void * __reserved1; /* placeholder for system variables */
void * __reserved2; /* placeholder for config options */
};
The st_mysql_plugin structure is common to
every type of plugin. Its members should be filled in as
follows:
typeThe plugin type. This must be one of the plugin-type values from
plugin.h. For a full-text parser plugin, thetypevalue isMYSQL_FTPARSER_PLUGIN.infoA pointer to the descriptor for the plugin. Unlike the general plugin declaration structure, this descriptor's structure depends on the particular type of plugin. Each descriptor has a version number that indicates the API version for that type of plugin, plus any other members needed. The descriptor for full-text plugins is described in Section 22.2.3.2, “Type-Specific Plugin Structures and Functions”.
nameThe plugin name. This is the name that will be listed in the
plugintable and by which you refer to the plugin in SQL statements such asINSTALL PLUGINandUNINSTALL PLUGIN.authorThe plugin author. This can be whatever you like.
descA general description of the plugin. This can be whatever you like.
The plugin license type. The value can be one of
PLUGIN_LICENSE_PROPRIETARY,PLUGIN_LICENSE_GPL, orPLUGIN_LICENSE_BSD.initA once-only initialization function. This is executed when the plugin is loaded, which happens for
INSTALL PLUGINor, for plugins listed in theplugintable, at server startup. The function takes no arguments. It returns zero for success and nonzero for failure. If aninitfunction is unneeded for a plugin, it can be specified as 0.deinitA once-only deinitialization function. This is executed when the plugin is unloaded, which happens for
UNINSTALL PLUGINor, for plugins listed in theplugintable, at server shutdown. The function takes no arguments. It returns zero for success and nonzero for failure. If adeinitfunction is unneeded for a plugin, it can be specified as 0.versionThe plugin version number. When the plugin is installed, this value can be retrieved from the
INFORMATION_SCHEMA.PLUGINStable. The value includes major and minor numbers. If you write the value as a hex constant, the format is0x, whereMMNNMMandNNare the major and minor numbers, respectively. For example,0x0302represents version 3.2.status_varsA pointer to a structure for status variables associated with the plugin, or 0 if there are no such variables. When the plugin is installed, these variables are displayed in the output of the
SHOW STATUSstatement.__reserved1,__reserved2These are placeholders for the future. Currently, they should be set to
NULL.
The init and deinit
functions in the general plugin declaration are invoked only
when loading and unloading the plugin. They have nothing to do
with use of the plugin such as happens when an SQL statement
causes the plugin to be invoked.
The status_vars member, if not 0, points to
an array of st_mysql_show_var structures,
each of which describes one status variable, followed by a
structure with all members set to 0. The
st_mysql_show_var structure has this
definition:
struct st_mysql_show_var {
const char *name;
char *value;
enum enum_mysql_show_type type;
};
When the plugin is installed, the plugin name and the
name value are joined with an underscore to
form the name displayed by SHOW
STATUS.
The following table shows the allowable status variable
type values and what the corresponding
variable should be.
| Type | Meaning |
SHOW_BOOL | Pointer to a boolean variable |
SHOW_INT | Pointer to an integer variable |
SHOW_LONG | Pointer to a long integer variable |
SHOW_LONGLONG | Pointer to a longlong integer variable |
SHOW_CHAR | A string |
SHOW_CHAR_PTR | Pointer to a string |
SHOW_ARRAY | Pointer to another st_mysql_show_var array |
SHOW_FUNC | Pointer to a function |
For the SHOW_FUNC type, the function is
called and fills in its out parameter,
which then provides information about the variable to be
displayed. The function has this signature:
#define SHOW_VAR_FUNC_BUFF_SIZE 1024
typedef int (*mysql_show_var_func) (void *thd,
struct st_mysql_show_var *out,
char *buf);
Plugins should consider the thd parameter
to be read only.
In the st_mysql_plugin structure that
defines a plugin's general declaration, the
info member points to a type-specific
plugin descriptor. For a full-text parser plugin, the
descriptor corresponds to the
st_mysql_ftparser structure in the
plugin.h file:
struct st_mysql_ftparser
{
int interface_version;
int (*parse)(MYSQL_FTPARSER_PARAM *param);
int (*init)(MYSQL_FTPARSER_PARAM *param);
int (*deinit)(MYSQL_FTPARSER_PARAM *param);
};
As shown by the structure definition, the descriptor has a
version number
(MYSQL_FTPARSER_INTERFACE_VERSION for
full-text parser plugins) and contains pointers to three
functions. The init and
deinit members should point to a function
or be set to 0 if the function is not needed. The
parse member must point to the function
that performs the parsing.
A full-text parser plugin is used in two different contexts, indexing and searching. In both contexts, the server calls the initialization and deinitialization functions at the beginning and end of processing each SQL statement that causes the plugin to be invoked. However, during statement processing, the server calls the main parsing function in context-specific fashion:
For indexing, the server calls the parser for each column value to be indexed.
For searching, the server calls the parser to parse the search string. The parser might also be called for rows processed by the statement. In natural language mode, there is no need for the server to call the parser. For boolean mode phrase searches or natural language searches with query expansion, the parser is used to parse column values for information that is not in the index. Also, if a boolean mode search is done for a column that has no
FULLTEXTindex, the built-in parser will be called. (Plugins are associated with specific indexes. If there is no index, no plugin is used.)
Note that the plugin declaration in the plugin library descriptor has initialization and deinitialization functions, and so does the plugin descriptor to which it points. These pairs of functions have different purposes and are invoked for different reasons:
For the plugin declaration in the plugin library descriptor, the initialization and deinitialization functions are invoked when the plugin is loaded and unloaded.
For the plugin descriptor, the initialization and deinitialization functions are invoked per SQL statement for which the plugin is used.
Each interface function named in the plugin descriptor should
return zero for success or nonzero for failure, and each of
them receives an argument that points to a
MYSQL_FTPARSER_PARAM structure containing
the parsing context. The structure has this definition:
typedef struct st_mysql_ftparser_param
{
int (*mysql_parse)(struct st_mysql_ftparser_param *,
char *doc, int doc_len);
int (*mysql_add_word)(struct st_mysql_ftparser_param *,
char *word, int word_len,
MYSQL_FTPARSER_BOOLEAN_INFO *boolean_info);
void *ftparser_state;
void *mysql_ftparam;
struct charset_info_st *cs;
char *doc;
int length;
int flags;
enum enum_ftparser_mode mode;
} MYSQL_FTPARSER_PARAM;
Note
The definition shown is current as of MySQL 5.1.12. It is incompatible with versions of MySQL 5.1 older than 5.1.12.
The structure members are used as follows:
mysql_parseA pointer to a callback function that invokes the server's built-in parser. Use this callback when the plugin acts as a front end to the built-in parser. That is, when the plugin parsing function is called, it should process the input to extract the text and pass the text to the
mysql_parsecallback.The first parameter for this callback function should be the
paramvalue itself:param->mysql_parse(param, ...);
A front end plugin can extract text and pass it all at once to the built-in parser, or it can extract and pass text to the built-in parser a piece at a time. However, in this case, the built-in parser treats the pieces of text as though there are implicit word breaks between them.
mysql_add_wordA pointer to a callback function that adds a word to a full-text index or to the list of search terms. Use this callback when the parser plugin replaces the built-in parser. That is, when the plugin parsing function is called, it should parse the input into words and invoke the
mysql_add_wordcallback for each word.The first parameter for this callback function should be the
paramvalue itself:param->mysql_add_word(param, ...);
ftparser_stateThis is a generic pointer. The plugin can set it to point to information to be used internally for its own purposes.
mysql_ftparamThis is set by the server. It is passed as the first argument to the
mysql_parseormysql_add_wordcallback.csA pointer to information about the character set of the text, or 0 if no information is available.
docA pointer to the text to be parsed.
lengthThe length of the text to be parsed, in bytes.
flagsParser flags. This is zero if there are no special flags. Currently, the only nonzero flag is
MYSQL_FTFLAGS_NEED_COPY, which means thatmysql_add_word()must save a copy of the word (that is, it cannot use a pointer to the word because the word is in a buffer that will be overwritten.) This member was added in MySQL 5.1.12.This flag might be set or reset by MySQL before calling the parser plugin, by the parser plugin itself, or by the
mysql_parse()function.modeThe parsing mode. This value will be one of the folowing constants:
MYSQL_FTPARSER_SIMPLE_MODEParse in fast and simple mode, which is used for indexing and for natural language queries. The parser should pass to the server only those words that should be indexed. If the parser uses length limits or a stopword list to determine which words to ignore, it should not pass such words to the server.
MYSQL_FTPARSER_WITH_STOPWORDSParse in stopword mode. This is used in boolean searches for phrase matching. The parser should pass all words to the server, even stopwords or words that are outside any normal length limits.
MYSQL_FTPARSER_FULL_BOOLEAN_INFOParse in boolean mode. This is used for parsing boolean query strings. The parser should recognize not only words but also boolean-mode operators and pass them to the server as tokens via the
mysql_add_wordcallback. To tell the server what kind of token is being passed, the plugin needs to fill in aMYSQL_FTPARSER_BOOLEAN_INFOstructure and pass a pointer to it.
If the parser is called in boolean mode, the
param->mode value will be
MYSQL_FTPARSER_FULL_BOOLEAN_INFO. The
MYSQL_FTPARSER_BOOLEAN_INFO structure that
the parser uses for passing token information to the server
looks like this:
typedef struct st_mysql_ftparser_boolean_info
{
enum enum_ft_token_type type;
int yesno;
int weight_adjust;
bool wasign;
bool trunc;
/* These are parser state and must be removed. */
byte prev;
byte *quot;
} MYSQL_FTPARSER_BOOLEAN_INFO;
The parser should fill in the structure members as follows:
typeThe token type. This should be one of values shown in the following table.
Type Meaning FT_TOKEN_EOFEnd of data FT_TOKEN_WORDA regular word FT_TOKEN_LEFT_PARENThe beginning of a group or subexpression FT_TOKEN_RIGHT_PARENThe end of a group or subexpression FT_TOKEN_STOPWORDA stopword yesnoWhether the word must be present for a match to occur. 0 means that the word is optional but increases the match relevance if it is present. Values larger than 0 mean that the word must be present. Values smaller than 0 mean that the word must not be present.
weight_adjustA weighting factor that determines how much a match for the word counts. It can be used to increase or decrease the word's importance in relevance calculations. A value of zero indicates no weight adjustment. Values greater than or less than zero mean higher or lower weight, respectively. The examples at Section 11.8.2, “Boolean Full-Text Searches”, that use the
<and>operators illustrate how weighting works.wasignThe sign of the weighting factor. A negative value acts like the
~boolean-search operator, which causes the word's contribution to the relevance to be negative.truncWhether matching should be done as if the boolean-mode
*truncation operator had been given.
Plugins should not use the prev and
quot members of the
MYSQL_FTPARSER_BOOLEAN_INFO structure.
This section provides a step-by-step procedure for creating a
plugin library. It shows how to develop a library that
contains a full-text parsing plugin named
simple_parser. This plugin performs parsing
based on simpler rules than those used by the MySQL built-in
full-text parser: Words are nonempty runs of whitespace
characters.
Each plugin library has the following contents:
A plugin library descriptor that indicates the version number of the general plugin API that the library uses and that contains a general declaration for each plugin in the library.
Each plugin general declaration contains information that is common to all types of plugin: A value that indicates the plugin type; the plugin name, author, description, and license type; and pointers to the initialization and deinitialization functions that the server invokes when it loads and unloads the plugin.
The plugin general declaration also contains a pointer to a type-specific plugin descriptor. The structure of these descriptors can vary from one plugin type to another, because each type of plugin can have its own API. A plugin descriptor contains a type-specific API version number and pointers to the functions that are needed to implement that plugin type. For example, a full-text parser plugin has initialization and deinitialization functions, and a main parsing function. The server invokes these functions when it uses the plugin to parse text.
The plugin library contains the interface functions that are referenced by the library descriptor and by the plugin descriptors.
The easiest way to follow the instructions in this section is
to use the source code in the
plugin/fulltext directory of a MySQL
source distribution. The instructions assume that you make a
copy of that directory and use it to build the plugin library.
To make a copy of the directory, use the following commands,
which assume that the MySQL source tree is in a directory
named mysql-5.1 under your
current directory:
shell>mkdir fulltext_pluginshell>cp mysql-5.1/plugin/fulltext/* fulltext_plugin
After copying the source files, use the following procedure to create a plugin library:
Change location into the
fulltext_plugindirectory:shell>
cd fulltext_pluginThe plugin source file should include the header files that the plugin library needs. The
plugin.hfile is required, and the library might require other files as well. For example:#include <stdlib.h> #include <ctype.h> #include <mysql/plugin.h>
Set up the plugin library file descriptor.
Every plugin library must include a library descriptor that must define two symbols:
_mysql_plugin_interface_version_specifies the version number of the general plugin framework. This is given by theMYSQL_PLUGIN_INTERFACE_VERSIONsymbol, which is defined in theplugin.hfile._mysql_plugin_declarations_defines an array of plugin declarations, terminated by a declaration with all members set to 0. Each declaration is an instance of thest_mysql_pluginstructure (also defined inplugin.h). There must be one of these for each plugin in the library.
If the server does not find these two symbols in a library, it does not accept it as a legal plugin library and rejects it with an error. This prevents use of a library for plugin purposes unless it was built specifically as a plugin library.
The standard (and most convenient) way to define the two required symbols is by using the
mysql_declare_pluginandmysql_declare_plugin_endmacros from theplugin.hfile:mysql_declare_plugin
... one or more plugin declarations here ...mysql_declare_plugin_end;For example, the library descriptor for a library that contains a single plugin named
simple_parserlooks like this:mysql_declare_plugin { MYSQL_FTPARSER_PLUGIN, /* type */ &simple_parser_descriptor, /* descriptor */ "simple_parser", /* name */ "MySQL AB", /* author */ "Simple Full-Text Parser", /* description */ PLUGIN_LICENSE_GPL, /* plugin license */ simple_parser_plugin_init, /* init function (when loaded) */ simple_parser_plugin_deinit,/* deinit function (when unloaded) */ 0x0001, /* version */ simple_status /* status variables */ } mysql_declare_plugin_end;For a full-text parser plugin, the type must be
MYSQL_FTPARSER_PLUGIN. This is the value that identifies the plugin as being legal for use in aWITH PARSERclause when creating aFULLTEXTindex. (No other plugin type is legal for this clause.)The
mysql_declare_pluginandmysql_declare_plugin_endmacros are defined inplugin.hlike this:#ifndef MYSQL_DYNAMIC_PLUGIN #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ int VERSION= MYSQL_PLUGIN_INTERFACE_VERSION; \ int PSIZE= sizeof(struct st_mysql_plugin); \ struct st_mysql_plugin DECLS[]= { #else #define __MYSQL_DECLARE_PLUGIN(NAME, VERSION, PSIZE, DECLS) \ int _mysql_plugin_interface_version_= MYSQL_PLUGIN_INTERFACE_VERSION; \ int _mysql_sizeof_struct_st_plugin_= sizeof(struct st_mysql_plugin); \ struct st_mysql_plugin _mysql_plugin_declarations_[]= { #endif #define mysql_declare_plugin(NAME) \ __MYSQL_DECLARE_PLUGIN(NAME, \ builtin_ ## NAME ## _plugin_interface_version, \ builtin_ ## NAME ## _sizeof_struct_st_plugin, \ builtin_ ## NAME ## _plugin) #define mysql_declare_plugin_end ,{0,0,0,0,0,0,0,0,0}}One point to note about those definitions is that the
_mysql_plugin_interface_version_symbol is defined only if theMYSQL_DYNAMIC_PLUGINsymbol is defined. This means that you'll need to provide-DMYSQL_DYNAMIC_PLUGINas part of the compilation command when you build the plugin.When the macros are used as just shown, they expand to the following code, which defines both of the required symbols (
_mysql_plugin_interface_version_and_mysql_plugin_declarations_):int _mysql_plugin_interface_version_= MYSQL_PLUGIN_INTERFACE_VERSION; struct st_mysql_plugin _mysql_plugin_declarations_[]= { { MYSQL_FTPARSER_PLUGIN, /* type */ &simple_parser_descriptor, /* descriptor */ "simple_parser", /* name */ "MySQL AB", /* author */ "Simple Full-Text Parser", /* description */ PLUGIN_LICENSE_GPL, /* plugin license */ simple_parser_plugin_init, /* init function (when loaded) */ simple_parser_plugin_deinit,/* deinit function (when unloaded) */ 0x0001, /* version */ simple_status /* status variables */ } ,{0,0,0,0,0,0,00,0} };The preceding example declares a single plugin in the library descriptor, but it is possible to declare multiple plugins. List the declarations one after the other between
mysql_declare_pluginandmysql_declare_plugin_end, separated by commas.MySQL plugins can be written in C or C++ (or another language that can use C calling conventions). One feature of C++ is that you can use nonconstant variables to initialize global structures. However, if you write a C++ plugin, you should not use this feature. Members of structures such as the
st_mysql_pluginstructure should be initialized with constant variables. See the discussion at the end of this section that describes some legal and illegal initializers for plugins.Set up the plugin descriptor.
Each plugin declaration in the library descriptor points to a type-specific descriptor for the corresponding plugin. In the
simple_parserdeclaration, that descriptor is indicated by&simple_parser_descriptor. The descriptor specifies the version number for the full-text plugin interface (as given byMYSQL_FTPARSER_INTERFACE_VERSION), and the plugin's parsing, initialization, and deinitialization functions:static struct st_mysql_ftparser simple_parser_descriptor= { MYSQL_FTPARSER_INTERFACE_VERSION, /* interface version */ simple_parser_parse, /* parsing function */ simple_parser_init, /* parser init function */ simple_parser_deinit /* parser deinit function */ };Set up the plugin interface functions.
The general plugin declaration in the library descriptor names the initialization and deinitialization functions that the server should invoke when it loads and unloads the plugin. For
simple_parser, these functions do nothing but return zero to indicate that they succeeded:static int simple_parser_plugin_init(void) { return(0); } static int simple_parser_plugin_deinit(void) { return(0); }Because those functions do not actually do anything, you could omit them and specify 0 for each of them in the plugin declaration.
The type-specific plugin descriptor for
simple_parsernames the initialization, deinitialization, and parsing functions that the server invokes when the plugin is used. Forsimple_parser, the initialization and deinitialization functions do nothing:static int simple_parser_init(MYSQL_FTPARSER_PARAM *param) { return(0); } static int simple_parser_deinit(MYSQL_FTPARSER_PARAM *param) { return(0); }Here too, because those functions do nothing, you could omit them and specify 0 for each of them in the plugin descriptor.
The main parsing function,
simple_parser_parse(), acts as a replacement for the built-in full-text parser, so it needs to split text into words and pass each word to the server. The parsing function's first argument is a pointer to a structure that contains the parsing context. This structure has adocmember that points to the text to be parsed, and alengthmember that indicates how long the text is. The simple parsing done by the plugin considers nonempty runs of whitespace characters to be words, so it identifies words like this:static int simple_parser_parse(MYSQL_FTPARSER_PARAM *param) { char *end, *start, *docend= param->doc + param->length; for (end= start= param->doc;; end++) { if (end == docend) { if (end > start) add_word(param, start, end - start); break; } else if (isspace(*end)) { if (end > start) add_word(param, start, end - start); start= end + 1; } } return(0); }As the parser finds each word, it invokes a function
add_word()to pass the word to the server.add_word()is a helper function only; it is not part of the plugin interface. The parser passes the parsing context pointer toadd_word(), as well as a pointer to the word and a length value:static void add_word(MYSQL_FTPARSER_PARAM *param, char *word, size_t len) { MYSQL_FTPARSER_BOOLEAN_INFO bool_info= { FT_TOKEN_WORD, 0, 0, 0, 0, ' ', 0 }; param->mysql_add_word(param, word, len, &bool_info); }For boolean-mode parsing,
add_word()fills in the members of thebool_infostructure as described in Section 22.2.3.2, “Type-Specific Plugin Structures and Functions”.Set up the status variables, if there are any. For the
simple_parserplugin, the following status variable array sets up one status variable with a value that is static text, and another with a value that is stored in a long integer variable:long number_of_calls= 0; struct st_mysql_show_var simple_status[]= { {"static", (char *)"just a static text", SHOW_CHAR}, {"called", (char *)&number_of_calls, SHOW_LONG}, {0,0,0} };When the plugin is installed, the plugin name and the
namevalue are joined with an underscore to form the name displayed bySHOW STATUS. For the array just shown, the resulting status variable names aresimple_parser_staticandsimple_parser_called. This convention means that you can easily display the variables for a plugin using its name:mysql>
SHOW STATUS LIKE 'simple_parser%';+----------------------+--------------------+ | Variable_name | Value | +----------------------+--------------------+ | simple_parser_static | just a static text | | simple_parser_called | 0 | +----------------------+--------------------+Compile the plugin library as a shared library and install it in the plugin directory.
Note
As mentioned earlier, be sure to specify
-DMYSQL_DYNAMIC_PLUGINas part of the compilation command when you build the plugin.The procedure for compiling shared objects varies from system to system. If you build your library using the GNU autotools, libtool should be able to generate the correct compilation commands for your system. If the library is named
mypluglib, you should end up with a shared object file that has a name something likelibmypluglib.so. (The file name might have a different extension on your system.)To use the autotools, you'll need to make a few changes to the configuration files at this point to enable the plugin to be compiled and installed. Assume that your MySQL distribution is installed at a base directory of
/usr/local/mysqland that its header files are located in theincludedirectory under the base directory.Edit
Makefile.am, which should look something like this:#Makefile.am example for a plugin pkglibdir=$(libdir)/mysql INCLUDES= -I$(top_builddir)/include -I$(top_srcdir)/include #noinst_LTLIBRARIES= mypluglib.la pkglib_LTLIBRARIES= mypluglib.la mypluglib_la_SOURCES= plugin_example.c mypluglib_la_LDFLAGS= -module -rpath $(pkglibdir) mypluglib_la_CFLAGS= -DMYSQL_DYNAMIC_PLUGIN
The
mypluglib_la_CFLAGSline takes care of passing the-DMYSQL_DYNAMIC_PLUGINflag to the compilation command.Adjust the
INCLUDESline to specify the path name to the installed MySQL header files. Edit it to look like this:INCLUDES= -I/usr/local/mysql/include
Make sure that the
noinst_LTLIBRARIESline is commented out or remove it. Make sure that thepkglib_LTLIBRARIESline is not commented out; it enables the make install command.Set up the files needed for the configure command, invoke it, and run make:
shell>
autoreconf --force --install --symlinkshell>./configure --prefix=/usr/local/mysqlshell>makeThe
--prefixoption to configure indicates the MySQL base directory under which the plugin should be installed. You can see what value to use for this option withSHOW VARIABLES:mysql>
SHOW VARIABLES LIKE 'basedir';+---------------+------------------+ | Variable_name | Value | +---------------+------------------+ | base | /usr/local/mysql | +---------------+------------------+The location of the plugin directory where you should install the library is given by the
plugin_dirsystem variable. For example:mysql>
SHOW VARIABLES LIKE 'plugin_dir';+---------------+-----------------------------------+ | Variable_name | Value | +---------------+-----------------------------------+ | plugin_dir | /usr/local/mysql/lib/mysql/plugin | +---------------+-----------------------------------+To install the plugin library, use make:
shell>
make installVerify that make install installed the plugin library in the proper directory. After installing it, make sure that the library permissions allow it to be executed by the server.
Register the plugin with the server.
The
INSTALL PLUGINstatement causes the server to list the plugin in theplugintable and to load the plugin code from the library file. Use that statement to registersimple_parserwith the server, and then verify that the plugin is listed in theplugintable:mysql>
INSTALL PLUGIN simple_parser SONAME 'libmypluglib.so';Query OK, 0 rows affected (0.00 sec) mysql>SELECT * FROM mysql.plugin;+---------------+-----------------+ | name | dl | +---------------+-----------------+ | simple_parser | libmypluglib.so | +---------------+-----------------+ 1 row in set (0.00 sec)Try the plugin.
Create a table that contains a string column and associate the parser plugin with a
FULLTEXTindex on the column:mysql>
CREATE TABLE t (c VARCHAR(255),->FULLTEXT (c) WITH PARSER simple_parser);Query OK, 0 rows affected (0.01 sec)Insert some text into the table and try some searches. These should verify that the parser plugin treats all nonwhitespace characters as word characters:
mysql>
INSERT INTO t VALUES->('latin1_general_cs is a case-sensitive collation'),->('I\'d like a case of oranges'),->('this is sensitive information'),->('another row'),->('yet another row');Query OK, 5 rows affected (0.02 sec) Records: 5 Duplicates: 0 Warnings: 0 mysql>SELECT c FROM t;+-------------------------------------------------+ | c | +-------------------------------------------------+ | latin1_general_cs is a case-sensitive collation | | I'd like a case of oranges | | this is sensitive information | | another row | | yet another row | +-------------------------------------------------+ 5 rows in set (0.00 sec) mysql>SELECT MATCH(c) AGAINST('case') FROM t;+--------------------------+ | MATCH(c) AGAINST('case') | +--------------------------+ | 0 | | 1.2968142032623 | | 0 | | 0 | | 0 | +--------------------------+ 5 rows in set (0.00 sec) mysql>SELECT MATCH(c) AGAINST('sensitive') FROM t;+-------------------------------+ | MATCH(c) AGAINST('sensitive') | +-------------------------------+ | 0 | | 0 | | 1.3253291845322 | | 0 | | 0 | +-------------------------------+ 5 rows in set (0.01 sec) mysql>SELECT MATCH(c) AGAINST('case-sensitive') FROM t;+------------------------------------+ | MATCH(c) AGAINST('case-sensitive') | +------------------------------------+ | 1.3109166622162 | | 0 | | 0 | | 0 | | 0 | +------------------------------------+ 5 rows in set (0.01 sec) mysql>SELECT MATCH(c) AGAINST('I\'d') FROM t;+--------------------------+ | MATCH(c) AGAINST('I\'d') | +--------------------------+ | 0 | | 1.2968142032623 | | 0 | | 0 | | 0 | +--------------------------+ 5 rows in set (0.01 sec)
Note how neither “case” nor “insensitive” match “case-insensitive” the way that they would for the built-in parser.
MySQL plugins can be written in C or C++ (or another language
that can use C calling conventions). One feature of C++ is
that you can use nonconstant variables to initialize global
structures. However, if you write a C++ plugin, you should not
use this feature. Members of structures such as the
st_mysql_plugin structure should be
initialized with constant variables. The
simple_parser descriptor shown earlier is
allowable in a C++ plugin because it satisfies that
requirement:
mysql_declare_plugin
{
MYSQL_FTPARSER_PLUGIN, /* type */
&simple_parser_descriptor, /* descriptor */
"simple_parser", /* name */
"MySQL AB", /* author */
"Simple Full-Text Parser", /* description */
PLUGIN_LICENSE_GPL, /* plugin license */
simple_parser_plugin_init, /* init function (when loaded) */
simple_parser_plugin_deinit,/* deinit function (when unloaded) */
0x0001, /* version */
simple_status /* status variables */
}
mysql_declare_plugin_end;
Here is another valid way to write the descriptor. It uses constant variables to indicate the plugin name, author, and description:
const char *simple_parser_name = "simple_parser";
const char *simple_parser_author = "MySQL AB";
const char *simple_parser_description = "Simple Full-Text Parser";
mysql_declare_plugin
{
MYSQL_FTPARSER_PLUGIN, /* type */
&simple_parser_descriptor, /* descriptor */
simple_parser_name, /* name */
simple_parser_author, /* author */
simple_parser_description, /* description */
PLUGIN_LICENSE_GPL, /* plugin license */
simple_parser_plugin_init, /* init function (when loaded) */
simple_parser_plugin_deinit,/* deinit function (when unloaded) */
0x0001, /* version */
simple_status /* status variables */
}
mysql_declare_plugin_end;
However, the following descriptor is invalid. It uses structure members to indicate the plugin name, author, and description, but structures are not considered constant initializers in C++:
typedef struct
{
const char *name;
const char *author;
const char *description;
} plugin_info;
plugin_info parser_info = {
"simple_parser",
"MySQL AB",
"Simple Full-Text Parser"
};
mysql_declare_plugin
{
MYSQL_FTPARSER_PLUGIN, /* type */
&simple_parser_descriptor, /* descriptor */
parser_info.name, /* name */
parser_info.author, /* author */
parser_info.description, /* description */
PLUGIN_LICENSE_GPL, /* plugin license */
simple_parser_plugin_init, /* init function (when loaded) */
simple_parser_plugin_deinit,/* deinit function (when unloaded) */
0x0001, /* version */
simple_status /* status variables */
}
mysql_declare_plugin_end;
There are three ways to add new functions to MySQL:
You can add functions through the user-defined function (UDF) interface. User-defined functions are compiled as object files and then added to and removed from the server dynamically using the
CREATE FUNCTIONandDROP FUNCTIONstatements. See Section 12.5.3.1, “CREATE FUNCTIONSyntax”.You can add functions as native (built-in) MySQL functions. Native functions are compiled into the mysqld server and become available on a permanent basis.
Another way to add functions is by creating stored functions. These are written using SQL statements rather than by compiling object code. The syntax for writing stored functions is not covered here. See Section 19.2, “Using Stored Routines (Procedures and Functions)”.
Each method of creating compiled functions has advantages and disadvantages:
If you write user-defined functions, you must install object files in addition to the server itself. If you compile your function into the server, you don't need to do that.
Native functions require you to modify a source distribution. UDFs do not. You can add UDFs to a binary MySQL distribution. No access to MySQL source is necessary.
If you upgrade your MySQL distribution, you can continue to use your previously installed UDFs, unless you upgrade to a newer version for which the UDF interface changes. For native functions, you must repeat your modifications each time you upgrade.
Whichever method you use to add new functions, they can be invoked
in SQL statements just like native functions such as
ABS() or
SOUNDEX().
See Section 8.2.4, “Function Name Parsing and Resolution”, for the rules describing how the server interprets references to different kinds of functions.
The following sections describe features of the UDF interface, provide instructions for writing UDFs, discuss security precautions that MySQL takes to prevent UDF misuse, and describe how to add native MySQL functions.
For example source code that illustrates how to write UDFs, take a
look at the sql/udf_example.c file that is
provided in MySQL source distributions.
The MySQL interface for user-defined functions provides the following features and capabilities:
Functions can return string, integer, or real values and can accept arguments of those same types.
You can define simple functions that operate on a single row at a time, or aggregate functions that operate on groups of rows.
Information is provided to functions that enables them to check the number, types, and names of the arguments passed to them.
You can tell MySQL to coerce arguments to a given type before passing them to a function.
You can indicate that a function returns
NULLor that an error occurred.
For the UDF mechanism to work, functions must be written in C or
C++ and your operating system must support dynamic loading. The
MySQL source distribution includes a file
sql/udf_example.c that defines 5 new
functions. Consult this file to see how UDF calling conventions
work. UDF-related symbols and data structures are defined in the
include/mysql_com.h header file. (You need
not include this header file directly because it is included by
mysql.h.)
A UDF contains code that becomes part of the running server, so
when you write a UDF, you are bound by any and all constraints
that otherwise apply to writing server code. For example, you
may have problems if you attempt to use functions from the
libstdc++ library. These constraints may
change in future versions of the server, so it is possible that
server upgrades will require revisions to UDFs that were
originally written for older servers. For information about
these constraints, see Section 2.10.2, “Typical configure Options”, and
Section 2.10.4, “Dealing with Problems Compiling MySQL”.
To be able to use UDFs, you need to link
mysqld dynamically. Don't configure MySQL
using --with-mysqld-ldflags=-all-static. If you
want to use a UDF that needs to access symbols from
mysqld (for example, the
metaphone function in
sql/udf_example.c that uses
default_charset_info), you must link the
program with -rdynamic (see man
dlopen). If you plan to use UDFs, the rule of thumb is
to configure MySQL with
--with-mysqld-ldflags=-rdynamic unless you have
a very good reason not to.
For each function that you want to use in SQL statements, you
should define corresponding C (or C++) functions. In the
following discussion, the name “xxx” is used for an
example function name. To distinguish between SQL and C/C++
usage, XXX() (uppercase) indicates an SQL
function call, and xxx() (lowercase)
indicates a C/C++ function call.
The C/C++ functions that you write to implement the interface
for XXX() are:
xxx()(required)The main function. This is where the function result is computed. The correspondence between the SQL function data type and the return type of your C/C++ function is shown here.
It is also possible to declare a
DECIMALfunction, but currently the value is returned as a string, so you should write the UDF as though it were aSTRINGfunction.ROWfunctions are not implemented.xxx_init()(optional)The initialization function for
xxx(). It can be used for the following purposes:To check the number of arguments to
XXX().To check that the arguments are of a required type or, alternatively, to tell MySQL to coerce arguments to the types you want when the main function is called.
To allocate any memory required by the main function.
To specify the maximum length of the result.
To specify (for
REALfunctions) the maximum number of decimal places in the result.To specify whether the result can be
NULL.
xxx_deinit()(optional)The deinitialization function for
xxx(). It should deallocate any memory allocated by the initialization function.
When an SQL statement invokes XXX(), MySQL
calls the initialization function xxx_init()
to let it perform any required setup, such as argument checking
or memory allocation. If xxx_init() returns
an error, MySQL aborts the SQL statement with an error message
and does not call the main or deinitialization functions.
Otherwise, MySQL calls the main function
xxx() once for each row. After all rows have
been processed, MySQL calls the deinitialization function
xxx_deinit() so that it can perform any
required cleanup.
For aggregate functions that work like
SUM(), you must also provide the
following functions:
xxx_clear()Reset the current aggregate value but do not insert the argument as the initial aggregate value for a new group.
xxx_add()Add the argument to the current aggregate value.
MySQL handles aggregate UDFs as follows:
Call
xxx_init()to let the aggregate function allocate any memory it needs for storing results.Sort the table according to the
GROUP BYexpression.Call
xxx_clear()for the first row in each new group.Call
xxx_add()for each row that belongs in the same group.Call
xxx()to get the result for the aggregate when the group changes or after the last row has been processed.Repeat 3-5 until all rows has been processed
Call
xxx_deinit()to let the UDF free any memory it has allocated.
All functions must be thread-safe. This includes not just the
main function, but the initialization and deinitialization
functions as well, and also the additional functions required by
aggregate functions. A consequence of this requirement is that
you are not allowed to allocate any global or static variables
that change! If you need memory, you should allocate it in
xxx_init() and free it in
xxx_deinit().
This section describes the different functions that you need to define when you create a simple UDF. Section 22.3.2, “Adding a New User-Defined Function”, describes the order in which MySQL calls these functions.
The main xxx() function should be declared
as shown in this section. Note that the return type and
parameters differ, depending on whether you declare the SQL
function XXX() to return
STRING,
INTEGER, or
REAL in the
CREATE FUNCTION statement:
For STRING functions:
char *xxx(UDF_INIT *initid, UDF_ARGS *args,
char *result, unsigned long *length,
char *is_null, char *error);
For INTEGER functions:
long long xxx(UDF_INIT *initid, UDF_ARGS *args,
char *is_null, char *error);
For REAL functions:
double xxx(UDF_INIT *initid, UDF_ARGS *args,
char *is_null, char *error);
DECIMAL functions return string
values and should be declared the same way as
STRING functions. ROW
functions are not implemented.
The initialization and deinitialization functions are declared like this:
my_bool xxx_init(UDF_INIT *initid, UDF_ARGS *args, char *message); void xxx_deinit(UDF_INIT *initid);
The initid parameter is passed to all three
functions. It points to a UDF_INIT
structure that is used to communicate information between
functions. The UDF_INIT structure members
follow. The initialization function should fill in any members
that it wishes to change. (To use the default for a member,
leave it unchanged.)
my_bool maybe_nullxxx_init()should setmaybe_nullto1ifxxx()can returnNULL. The default value is1if any of the arguments are declaredmaybe_null.unsigned int decimalsThe number of decimal digits to the right of the decimal point. The default value is the maximum number of decimal digits in the arguments passed to the main function. For example, if the function is passed
1.34,1.345, and1.3, the default would be 3, because1.345has 3 decimal digits.For arguments that have no fixed number of decimals, the
decimalsvalue is set to 31, which is 1 more than the maximum number of decimals allowed for theDECIMAL,FLOAT, andDOUBLEdata types.A
decimalsvalue of 31 is used for arguments in cases such as aFLOATorDOUBLEcolumn declared without an explicit number of decimals (for example,FLOATrather thanFLOAT(10,3)) and for floating-point constants such as1345E-3. It is also used for string and other nonnumber arguments that might be converted within the function to numeric form.The value to which the
decimalsmember is initialized is only a default. It can be changed within the function to reflect the actual calculation performed. The default is determined such that the largest number of decimals of the arguments is used. If the number of decimals is 31 for even one of the arguments, that is the value used fordecimals.unsigned int max_lengthThe maximum length of the result. The default
max_lengthvalue differs depending on the result type of the function. For string functions, the default is the length of the longest argument. For integer functions, the default is 21 digits. For real functions, the default is 13 plus the number of decimal digits indicated byinitid->decimals. (For numeric functions, the length includes any sign or decimal point characters.)If you want to return a blob value, you can set
max_lengthto 65KB or 16MB. This memory is not allocated, but the value is used to decide which data type to use if there is a need to temporarily store the data.char *ptrA pointer that the function can use for its own purposes. For example, functions can use
initid->ptrto communicate allocated memory among themselves.xxx_init()should allocate the memory and assign it to this pointer:initid->ptr = allocated_memory;
In
xxx()andxxx_deinit(), refer toinitid->ptrto use or deallocate the memory.my_bool const_itemxxx_init()should setconst_itemto1ifxxx()always returns the same value and to0otherwise.
This section describes the different functions that you need to define when you create an aggregate UDF. Section 22.3.2, “Adding a New User-Defined Function”, describes the order in which MySQL calls these functions.
xxx_reset()This function is called when MySQL finds the first row in a new group. It should reset any internal summary variables and then use the given
UDF_ARGSargument as the first value in your internal summary value for the group. Declarexxx_reset()as follows:void xxx_reset(UDF_INIT *initid, UDF_ARGS *args, char *is_null, char *error);xxx_reset()is not needed or used in MySQL 5.1, in which the UDF interface usesxxx_clear()instead. However, you can define bothxxx_reset()andxxx_clear()if you want to have your UDF work with older versions of the server. (If you do include both functions, thexxx_reset()function in many cases can be implemented internally by callingxxx_clear()to reset all variables, and then callingxxx_add()to add theUDF_ARGSargument as the first value in the group.)xxx_clear()This function is called when MySQL needs to reset the summary results. It is called at the beginning for each new group but can also be called to reset the values for a query where there were no matching rows. Declare
xxx_clear()as follows:void xxx_clear(UDF_INIT *initid, char *is_null, char *error);
is_nullis set to point toCHAR(0)before callingxxx_clear().If something went wrong, you can store a value in the variable to which the
errorargument points.errorpoints to a single-byte variable, not to a string buffer.xxx_clear()is required by MySQL 5.1.xxx_add()This function is called for all rows that belong to the same group. You should use it to add the value in the
UDF_ARGSargument to your internal summary variable.void xxx_add(UDF_INIT *initid, UDF_ARGS *args, char *is_null, char *error);
The xxx() function for an aggregate UDF
should be declared the same way as for a nonaggregate UDF. See
Section 22.3.2.1, “UDF Calling Sequences for Simple Functions”.
For an aggregate UDF, MySQL calls the xxx()
function after all rows in the group have been processed. You
should normally never access its UDF_ARGS
argument here but instead return a value based on your
internal summary variables.
Return value handling in xxx() should be
done the same way as for a nonaggregate UDF. See
Section 22.3.2.4, “UDF Return Values and Error Handling”.
The xxx_reset() and
xxx_add() functions handle their
UDF_ARGS argument the same way as functions
for nonaggregate UDFs. See Section 22.3.2.3, “UDF Argument Processing”.
The pointer arguments to is_null and
error are the same for all calls to
xxx_reset(),
xxx_clear(), xxx_add()
and xxx(). You can use this to remember
that you got an error or whether the xxx()
function should return NULL. You should not
store a string into *error!
error points to a single-byte variable, not
to a string buffer.
*is_null is reset for each group (before
calling xxx_clear()).
*error is never reset.
If *is_null or *error
are set when xxx() returns, MySQL returns
NULL as the result for the group function.
The args parameter points to a
UDF_ARGS structure that has the members
listed here:
unsigned int arg_countThe number of arguments. Check this value in the initialization function if you require your function to be called with a particular number of arguments. For example:
if (args->arg_count != 2) { strcpy(message,"XXX() requires two arguments"); return 1; }For other
UDF_ARGSmember values that are arrays, array references are zero-based. That is, refer to array members using index values from 0 toargs->arg_count– 1.enum Item_result *arg_typeA pointer to an array containing the types for each argument. The possible type values are
STRING_RESULT,INT_RESULT,REAL_RESULT, andDECIMAL_RESULT.To make sure that arguments are of a given type and return an error if they are not, check the
arg_typearray in the initialization function. For example:if (args->arg_type[0] != STRING_RESULT || args->arg_type[1] != INT_RESULT) { strcpy(message,"XXX() requires a string and an integer"); return 1; }Arguments of type
DECIMAL_RESULTare passed as strings, so you should handle them the same way asSTRING_RESULTvalues.As an alternative to requiring your function's arguments to be of particular types, you can use the initialization function to set the
arg_typeelements to the types you want. This causes MySQL to coerce arguments to those types for each call toxxx(). For example, to specify that the first two arguments should be coerced to string and integer, respectively, do this inxxx_init():args->arg_type[0] = STRING_RESULT; args->arg_type[1] = INT_RESULT;
Exact-value decimal arguments such as
1.3orDECIMALcolumn values are passed with a type ofDECIMAL_RESULT. However, the values are passed as strings. If you want to receive a number, use the initialization function to specify that the argument should be coerced to aREAL_RESULTvalue:args->arg_type[2] = REAL_RESULT;
char **argsargs->argscommunicates information to the initialization function about the general nature of the arguments passed to your function. For a constant argumenti,args->args[i]points to the argument value. (See below for instructions on how to access the value properly.) For a nonconstant argument,args->args[i]is0. A constant argument is an expression that uses only constants, such as3or4*7-2orSIN(3.14). A nonconstant argument is an expression that refers to values that may change from row to row, such as column names or functions that are called with nonconstant arguments.For each invocation of the main function,
args->argscontains the actual arguments that are passed for the row currently being processed.If argument
irepresentsNULL,args->args[i]is a null pointer (0). If the argument is notNULL, functions can refer to it as follows:An argument of type
STRING_RESULTis given as a string pointer plus a length, to allow handling of binary data or data of arbitrary length. The string contents are available asargs->args[i]and the string length isargs->lengths[i]. Do not assume that the string is null-terminated.For an argument of type
INT_RESULT, you must castargs->args[i]to along longvalue:long long int_val; int_val = *((long long*) args->args[i]);
For an argument of type
REAL_RESULT, you must castargs->args[i]to adoublevalue:double real_val; real_val = *((double*) args->args[i]);
For an argument of type
DECIMAL_RESULT, the value is passed as a string and should be handled like aSTRING_RESULTvalue.ROW_RESULTarguments are not implemented.
unsigned long *lengthsFor the initialization function, the
lengthsarray indicates the maximum string length for each argument. You should not change these. For each invocation of the main function,lengthscontains the actual lengths of any string arguments that are passed for the row currently being processed. For arguments of typesINT_RESULTorREAL_RESULT,lengthsstill contains the maximum length of the argument (as for the initialization function).char *maybe_nullFor the initialization function, the
maybe_nullarray indicates for each argument whether the argument value might be null (0 if no, 1 if yes).char **attributesargs->attributescommunicates information about the names of the UDF arguments. For argumenti, the attribute name is available as a string inargs->attributes[i]and the attribute length isargs->attribute_lengths[i]. Do not assume that the string is null-terminated.By default, the name of a UDF argument is the text of the expression used to specify the argument. For UDFs, an argument may also have an optional
[AS]clause, in which case the argument name isalias_namealias_name. Theattributesvalue for each argument thus depends on whether an alias was given.Suppose that a UDF
my_udf()is invoked as follows:SELECT my_udf(expr1, expr2 AS alias1, expr3 alias2);
In this case, the
attributesandattribute_lengthsarrays will have these values:args->attributes[0] = "expr1" args->attribute_lengths[0] = 5 args->attributes[1] = "alias1" args->attribute_lengths[1] = 6 args->attributes[2] = "alias2" args->attribute_lengths[2] = 6
unsigned long *attribute_lengthsThe
attribute_lengthsarray indicates the length of each argument name.
The initialization function should return 0
if no error occurred and 1 otherwise. If an
error occurs, xxx_init() should store a
null-terminated error message in the
message parameter. The message is returned
to the client. The message buffer is
MYSQL_ERRMSG_SIZE characters long, but you
should try to keep the message to less than 80 characters so
that it fits the width of a standard terminal screen.
The return value of the main function xxx()
is the function value, for long long and
double functions. A string function should
return a pointer to the result and set
*length to the length (in bytes) of the
return value. For example:
memcpy(result, "result string", 13); *length = 13;
MySQL passes a buffer to the xxx() function
via the result parameter. This buffer is
sufficiently long to hold 255 characters, which can be
multi-byte characters. The xxx() function
can store the result in this buffer if it fits, in which case
the return value should be a pointer to the buffer. If the
function stores the result in a different buffer, it should
return a pointer to that buffer.
If your string function does not use the supplied buffer (for
example, if it needs to return a string longer than 255
characters), you must allocate the space for your own buffer
with malloc() in your
xxx_init() function or your
xxx() function and free it in your
xxx_deinit() function. You can store the
allocated memory in the ptr slot in the
UDF_INIT structure for reuse by future
xxx() calls. See
Section 22.3.2.1, “UDF Calling Sequences for Simple Functions”.
To indicate a return value of NULL in the
main function, set *is_null to
1:
*is_null = 1;
To indicate an error return in the main function, set
*error to 1:
*error = 1;
If xxx() sets *error to
1 for any row, the function value is
NULL for the current row and for any
subsequent rows processed by the statement in which
XXX() was invoked.
(xxx() is not even called for subsequent
rows.)
Files implementing UDFs must be compiled and installed on the
host where the server runs. This process is described below
for the example UDF file
sql/udf_example.c that is included in the
MySQL source distribution.
The immediately following instructions are for Unix. Instructions for Windows are given later in this section.
The udf_example.c file contains the
following functions:
metaphon()returns a metaphon string of the string argument. This is something like a soundex string, but it is more tuned for English.myfunc_double()returns the sum of the ASCII values of the characters in its arguments, divided by the sum of the length of its arguments.myfunc_int()returns the sum of the length of its arguments.sequence([const int])returns a sequence starting from the given number or 1 if no number has been given.lookup()returns the IP number for a host name.reverse_lookup()returns the host name for an IP number. The function may be called either with a single string argument of the form'xxx.xxx.xxx.xxx'or with four numbers.
A dynamically loadable file should be compiled as a sharable object file, using a command something like this:
shell> gcc -shared -o udf_example.so udf_example.c
If you are using gcc with
configure and libtool
(which is how MySQL is configured), you should be able to
create udf_example.so with a simpler
command:
shell> make udf_example.la
After you compile a shared object containing UDFs, you must
install it and tell MySQL about it. Compiling a shared object
from udf_example.c using
gcc directly produces a file named
udf_example.so. Compiling the shared
object using make produces a file named
something like udf_example.so.0.0.0 in
the .libs directory (the exact name may
vary from platform to platform). Copy the shared object to the
server's plugin directory and name it
udf_example.so. This directory is given
by the value of the
plugin_dir system variable.
Note
This is a change in MySQL 5.1. For earlier versions of MySQL, the shared object can be located in any directory that is searched by your system's dynamic linker.
On some systems, the ldconfig program that
configures the dynamic linker does not recognize a shared
object unless its name begins with lib. In
this case you should rename a file such as
udf_example.so to
libudf_example.so.
On Windows, you can compile user-defined functions by using the following procedure:
You need to obtain the Bazaar source repository for MySQL 5.1. See Section 2.10.3, “Installing from the Development Source Tree”.
You must obtain the CMake build utility from http://www.cmake.org. (Version 2.4.2 or later is required).
In the source repository, look in the
sqldirectory. There are files namedudf_example.defudf_example.cthere. Copy both files from this directory to your working directory.Create a CMake
makefile(CMakeLists.txt) with these contents:PROJECT(udf_example) # Path for MySQL include directory INCLUDE_DIRECTORIES("c:/mysql/include") ADD_DEFINITIONS("-DHAVE_DLOPEN") ADD_LIBRARY(udf_example MODULE udf_example.c udf_example.def) TARGET_LINK_LIBRARIES(udf_example wsock32)Create the VC project and solution files:
cmake -G "<Generator>"
Invoking cmake --help shows you a list of valid Generators.
Create
udf_example.dll:devenv udf_example.sln /build Release
After the shared object file has been installed, notify mysqld about the new functions with these statements:
mysql>CREATE FUNCTION metaphon RETURNS STRING SONAME 'udf_example.dll';mysql>CREATE FUNCTION myfunc_double RETURNS REAL SONAME 'udf_example.dll';mysql>CREATE FUNCTION myfunc_int RETURNS INTEGER SONAME 'udf_example.dll';mysql>CREATE FUNCTION sequence RETURNS INTEGER SONAME 'udf_example.dll';mysql>CREATE FUNCTION lookup RETURNS STRING SONAME 'udf_example.dll';mysql>CREATE FUNCTION reverse_lookup->RETURNS STRING SONAME 'udf_example.dll';mysql>CREATE AGGREGATE FUNCTION avgcost->RETURNS REAL SONAME 'udf_example.dll';
Functions can be deleted using DROP
FUNCTION:
mysql>DROP FUNCTION metaphon;mysql>DROP FUNCTION myfunc_double;mysql>DROP FUNCTION myfunc_int;mysql>DROP FUNCTION sequence;mysql>DROP FUNCTION lookup;mysql>DROP FUNCTION reverse_lookup;mysql>DROP FUNCTION avgcost;
The CREATE FUNCTION and
DROP FUNCTION statements update
the func system table in the
mysql database. The function's name, type
and shared library name are saved in the table. You must have
the INSERT and
DELETE privileges for the
mysql database to create and drop
functions.
You should not use CREATE
FUNCTION to add a function that has previously been
created. If you need to reinstall a function, you should
remove it with DROP FUNCTION
and then reinstall it with CREATE
FUNCTION. You would need to do this, for example, if
you recompile a new version of your function, so that
mysqld gets the new version. Otherwise, the
server continues to use the old version.
An active function is one that has been loaded with
CREATE FUNCTION and not removed
with DROP FUNCTION. All active
functions are reloaded each time the server starts, unless you
start mysqld with the
--skip-grant-tables option. In
this case, UDF initialization is skipped and UDFs are
unavailable.
If the new function will be referred to in statements that will be replicated to slave servers, you must ensure that every slave server also has the function available. Otherwise, replication will fail on the slaves when they attempt to invoke the function.
MySQL takes the following measures to prevent misuse of user-defined functions.
You must have the INSERT
privilege to be able to use CREATE
FUNCTION and the
DELETE privilege to be able to
use DROP FUNCTION. This is
necessary because these statements add and delete rows from
the mysql.func table.
UDFs should have at least one symbol defined in addition to
the xxx symbol that corresponds to the main
xxx() function. These auxiliary symbols
correspond to the xxx_init(),
xxx_deinit(),
xxx_reset(),
xxx_clear(), and
xxx_add() functions.
mysqld also supports an
--allow-suspicious-udfs option
that controls whether UDFs that have only an
xxx symbol can be loaded. By default, the
option is off, to prevent attempts at loading functions from
shared object files other than those containing legitimate
UDFs. If you have older UDFs that contain only the
xxx symbol and that cannot be recompiled to
include an auxiliary symbol, it may be necessary to specify
the --allow-suspicious-udfs
option. Otherwise, you should avoid enabling this capability.
UDF object files cannot be placed in arbitrary directories.
They must be located in the server's plugin directory. This
directory is given by the value of the
plugin_dir system variable.
Note
This is a change in MySQL 5.1. For earlier versions of MySQL, the shared object can be located in any directory that is searched by your system's dynamic linker.
To add a new native MySQL function, use the procedure described here, which requires that you use a source distribution. You cannot add native functions to a binary distribution because it is necessary to modify MySQL source code and compile MySQL from the modified source. If you migrate to another version of MySQL (for example, when a new version is released), you must repeat the procedure with the new version.
If the new native function will be referred to in statements that will be replicated to slave servers, you must ensure that every slave server also has the function available. Otherwise, replication will fail on the slaves when they attempt to invoke the function.
To add a new native function, follow these steps to modify
source files in the sql directory. For
MySQL 5.1, the first two steps apply only as of 5.1.13. For
older versions, see the instructions in the corresponding
section of the MySQL 5.0 manual.
Create a subclass for the function in
item_create.cc:If the function takes a fixed number of arguments, create a subclass of
Create_func_arg0,Create_func_arg1,Create_func_arg2, orCreate_func_arg3, respectively, depending on whether the function takes zero, one, two, or three arguments. For examples, see theCreate_func_uuid,Create_func_abs,Create_func_pow, andCreate_func_lpadclasses.If the function takes a variable number of arguments, create a subclass of
Create_native_func. For an example, seeCreate_func_concat.
To provide a name by which the function can be referred to in SQL statements, register the name in
item_create.ccby adding a line to this array:static Native_func_registry func_array[]
You can register several names for the same function. For example, see the lines for
"LCASE"and"LOWER", which are aliases forCreate_func_lcase.In
item_func.h, declare a class inheriting fromItem_num_funcorItem_str_func, depending on whether your function returns a number or a string.In
item_func.cc, add one of the following declarations, depending on whether you are defining a numeric or string function:double Item_func_newname::val() longlong Item_func_newname::val_int() String *Item_func_newname::Str(String *str)
If you inherit your object from any of the standard items (like
Item_num_func), you probably only have to define one of these functions and let the parent object take care of the other functions. For example, theItem_str_funcclass defines aval()function that executesatof()on the value returned by::str().If the function is nondeterministic, include the following statement in the item constructor to indicate that function results should not be cached:
current_thd->lex->safe_to_cache_query=0;
A function is nondeterministic if, given fixed values for its arguments, it can return different results for different invocations.
You should probably also define the following object function:
void Item_func_newname::fix_length_and_dec()
This function should at least calculate
max_lengthbased on the given arguments.max_lengthis the maximum number of characters the function may return. This function should also setmaybe_null = 0if the main function can't return aNULLvalue. The function can check whether any of the function arguments can returnNULLby checking the arguments'maybe_nullvariable. Look atItem_func_mod::fix_length_and_decfor a typical example of how to do this.
All functions must be thread-safe. In other words, do not use any global or static variables in the functions without protecting them with mutexes.
If you want to return NULL from
::val(), ::val_int(), or
::str(), you should set
null_value to 1 and return 0.
For ::str() object functions, there are
additional considerations to be aware of:
The
String *strargument provides a string buffer that may be used to hold the result. (For more information about theStringtype, take a look at thesql_string.hfile.)The
::str()function should return the string that holds the result, or(char*) 0if the result isNULL.All current string functions try to avoid allocating any memory unless absolutely necessary!
In MySQL, you can define a procedure in C++ that can access and
modify the data in a query before it is sent to the client. The
modification can be done on a row-by-row or GROUP
BY level.
We have created an example procedure to show you what can be done.
Additionally, it is a good idea to take a look at
mylua. With this you can use the LUA language
to load a procedure at runtime into mysqld.
analyse([
max_elements[,max_memory]])
This procedure is defined in the
sql/sql_analyse.cc file. It examines the
result from a query and returns an analysis of the results that
suggests optimal data types for each column. To obtain this
analysis, append PROCEDURE ANALYSE to the end
of a SELECT statement:
SELECT ... FROM ... WHERE ... PROCEDURE ANALYSE([max_elements,[max_memory]])
For example:
SELECT col1, col2 FROM table1 PROCEDURE ANALYSE(10, 2000);
The results show some statistics for the values returned by the
query, and propose an optimal data type for the columns. This
can be helpful for checking your existing tables, or after
importing new data. You may need to try different settings for
the arguments so that PROCEDURE ANALYSE()
does not suggest the ENUM data
type when it is not appropriate.
The arguments are optional and are used as follows:
max_elements(default 256) is the maximum number of distinct values thatanalysenotices per column. This is used byanalyseto check whether the optimal data type should be of typeENUM; if there are more thanmax_elementsdistinct values, thenENUMis not a suggested type.max_memory(default 8192) is the maximum amount of memory thatanalyseshould allocate per column while trying to find all distinct values.
This appendix helps you port MySQL to other operating systems. Do check the list of currently supported operating systems first. See Section 2.1.1, “Operating Systems Supported by MySQL Community Server”. If you have created a new port of MySQL, please let us know so that we can list it here and on our Web site (http://www.mysql.com/), recommending it to other users.
Note: If you create a new port of MySQL, you are free to copy and distribute it under the GPL license, but it does not make you a copyright holder of MySQL.
A working POSIX thread library is needed for the server. On
Solaris 2.5 we use Sun PThreads (the native thread support in 2.4
and earlier versions is not good enough), on Linux we use
LinuxThreads by Xavier Leroy,
<Xavier.Leroy@inria.fr>.
The hard part of porting to a new Unix variant without good native
thread support is probably to port MIT-pthreads. See
mit-pthreads/README and Programming POSIX
Threads (http://www.humanfactor.com/pthreads/).
Up to MySQL 4.0.2, the MySQL distribution included a patched version of Chris Provenzano's Pthreads from MIT (see the MIT Pthreads Web page at http://www.mit.edu/afs/sipb/project/pthreads/ and a programming introduction at http://www.mit.edu:8001/people/proven/IAP_2000/). These can be used for some operating systems that do not have POSIX threads. See Section 2.10.5, “MIT-pthreads Notes”.
It is also possible to use another user level thread package named FSU Pthreads (see http://moss.csc.ncsu.edu/~mueller/pthreads/). This implementation is being used for the SCO port.
See the thr_lock.c and
thr_alarm.c programs in the
mysys directory for some tests/examples of
these problems.
Both the server and the client need a working C++ compiler. We use gcc on many platforms. Other compilers that are known to work are SPARCworks, Sun Forte, Irix cc, HP-UX aCC, IBM AIX xlC_r), Intel ecc/icc and Compaq cxx).
Important
If you are trying to build MySQL 5.1 with icc on the IA64 platform, and need support for MySQL Cluster, you should first ensure that you are using icc version 9.1.043 or later. (For details, see Bug#21875.)
To compile only the client use ./configure --without-server.
There is currently no support for only compiling the server, nor is it likely to be added unless someone has a good reason for it.
If you want/need to change any Makefile or
the configure script you also need GNU Automake and Autoconf. See
Section 2.10.3, “Installing from the Development Source Tree”.
All steps needed to remake everything from the most basic files.
/bin/rm */.deps/*.P /bin/rm -f config.cache aclocal autoheader aclocal automake autoconf ./configure --with-debug=full --prefix='your installation directory' # The makefiles generated above need GNU make 3.75 or newer. # (called gmake below) gmake clean all install init-db
If you run into problems with a new port, you may have to do some debugging of MySQL! See Section 22.5.1, “Debugging a MySQL Server”.
Note
Before you start debugging mysqld, first get
the test programs mysys/thr_alarm and
mysys/thr_lock to work. This ensures that
your thread installation has even a remote chance to work!
- 22.5.1.1. Compiling MySQL for Debugging
- 22.5.1.2. Creating Trace Files
- 22.5.1.3. Using
pdbto create a Windows crashdump - 22.5.1.4. Debugging mysqld under gdb
- 22.5.1.5. Using a Stack Trace
- 22.5.1.6. Using Server Logs to Find Causes of Errors in mysqld
- 22.5.1.7. Making a Test Case If You Experience Table Corruption
If you are using some functionality that is very new in MySQL,
you can try to run mysqld with the
--skip-new (which disables all new, potentially
unsafe functionality) or with
--safe-mode which disables a lot
of optimization that may cause problems. See
Section B.1.4.2, “What to Do If MySQL Keeps Crashing”.
If mysqld doesn't want to start, you should
verify that you don't have any my.cnf files
that interfere with your setup! You can check your
my.cnf arguments with mysqld
--print-defaults and avoid using them by starting with
mysqld --no-defaults ....
If mysqld starts to eat up CPU or memory or if it “hangs,” you can use mysqladmin processlist status to find out if someone is executing a query that takes a long time. It may be a good idea to run mysqladmin -i10 processlist status in some window if you are experiencing performance problems or problems when new clients can't connect.
The command mysqladmin debug dumps some information about locks in use, used memory and query usage to the MySQL log file. This may help solve some problems. This command also provides some useful information even if you haven't compiled MySQL for debugging!
If the problem is that some tables are getting slower and slower
you should try to optimize the table with
OPTIMIZE TABLE or
myisamchk. See
Chapter 5, MySQL Server Administration. You should also check
the slow queries with EXPLAIN.
You should also read the OS-specific section in this manual for problems that may be unique to your environment. See Section 2.13, “Operating System-Specific Notes”.
If you have some very specific problem, you can always try to
debug MySQL. To do this you must configure MySQL with the
--with-debug or the
--with-debug=full option.
You can check whether MySQL was compiled with debugging by
doing: mysqld --help. If the
--debug flag is listed with the
options then you have debugging enabled. mysqladmin
ver also lists the mysqld version
as mysql ... --debug in this case.
If you are using gcc, the recommended configure line is:
CC=gcc CFLAGS="-O2" CXX=gcc CXXFLAGS="-O2 -felide-constructors \ -fno-exceptions -fno-rtti" ./configure --prefix=/usr/local/mysql \ --with-debug --with-extra-charsets=complex
This avoids problems with the libstdc++
library and with C++ exceptions (many compilers have problems
with C++ exceptions in threaded code) and compile a MySQL
version with support for all character sets.
If you suspect a memory overrun error, you can configure MySQL
with --with-debug=full,
which installs a memory allocation
(SAFEMALLOC) checker. However, running with
SAFEMALLOC is quite slow, so if you get
performance problems you should start
mysqld with the
--skip-safemalloc option. This
disables the memory overrun checks for each call to
malloc() and free().
If mysqld stops crashing when you compile
it with --with-debug, you
probably have found a compiler bug or a timing bug within
MySQL. In this case, you can try to add -g to
the CFLAGS and CXXFLAGS
variables above and not use
--with-debug. If
mysqld dies, you can at least attach to it
with gdb or use gdb on
the core file to find out what happened.
When you configure MySQL for debugging you automatically
enable a lot of extra safety check functions that monitor the
health of mysqld. If they find something
“unexpected,” an entry is written to
stderr, which
mysqld_safe directs to the error log! This
also means that if you are having some unexpected problems
with MySQL and are using a source distribution, the first
thing you should do is to configure MySQL for debugging! (The
second thing is to send mail to a MySQL mailing list and ask
for help. See Section 1.5.1, “MySQL Mailing Lists”. If you believe
that you have found a bug, please use the instructions at
Section 1.6, “How to Report Bugs or Problems”.
In the Windows MySQL distribution,
mysqld.exe is by default compiled with
support for trace files.
If the mysqld server doesn't start or if you can cause it to crash quickly, you can try to create a trace file to find the problem.
To do this, you must have a mysqld that has
been compiled with debugging support. You can check this by
executing mysqld -V. If the version number
ends with -debug, it is compiled with
support for trace files. (On Windows, the debugging server is
named mysqld-debug rather than
mysqld as of MySQL 4.1.)
Start the mysqld server with a trace log in
/tmp/mysqld.trace on Unix or
C:\mysqld.trace on Windows:
shell> mysqld --debug
On Windows, you should also use the
--standalone flag to not start
mysqld as a service. In a console window,
use this command:
C:\> mysqld-debug --debug --standalone
After this, you can use the mysql.exe
command-line tool in a second console window to reproduce the
problem. You can stop the mysqld server
with mysqladmin shutdown.
Note that the trace file become very big! If you want to generate a smaller trace file, you can use debugging options something like this:
mysqld --debug=d,info,error,query,general,where:O,/tmp/mysqld.trace
This only prints information with the most interesting tags to the trace file.
If you make a bug report about this, please only send the lines from the trace file to the appropriate mailing list where something seems to go wrong! If you can't locate the wrong place, you can ftp the trace file, together with a full bug report, to ftp://ftp.mysql.com/pub/mysql/upload/ so that a MySQL developer can take a look at it.
The trace file is made with the DBUG package by Fred Fish. See Section 22.5.3, “The DBUG Package”.
Starting with MySQL 5.1.12 the Program Database files
(extension pdb) are included in the
Noinstall distribution of MySQL. These files provide
information for debugging your MySQL installation in the event
of a problem.
The PDB file contains more detailed information about
mysqld and other tools that enables more
detailed trace and dump files to be created. You can use these
with Dr Watson, WinDbg and Visual Studio to
debug mysqld.
For more information on PDB files, see Microsoft Knowledge Base Article 121366. For more information on the debugging options available, see Debugging Tools for Windows.
Dr Watson is installed with all Windows distributions, but if you have installed Windows development tools, Dr Watson may have been replaced with WinDbg, the debugger included with Visual Studio, or the debugging tools provided with Borland or Delphi.
To generate a crash file using Dr Watson, follow these steps:
Start Dr Watson by running drwtsn32.exe interactively using the
-ioption:C:\> drwtsn32 -i
Set the Log File Path to the directory where you want to store trace files.
Make sure Dump All Thread Contexts and Append To Existing Log File.
Uncheck Dump Sumbol Table, Visual Notification, Sound Notification and Create Crash Dump File.
Set the Number of Instructions to a suitable value to capture enough calls in the stacktrace. A value of at 25 should be enough.
Note that the file generated can be very large.
On most systems you can also start mysqld from gdb to get more information if mysqld crashes.
With some older gdb versions on Linux you
must use run --one-thread if you want to be
able to debug mysqld threads. In this case,
you can only have one thread active at a time. It is best to
upgrade to gdb 5.1 because thread debugging
works much better with this version!
NPTL threads (the new thread library on Linux) may cause problems while running mysqld under gdb. Some symptoms are:
In this case, you should set the following environment variable in the shell before starting gdb:
LD_ASSUME_KERNEL=2.4.1 export LD_ASSUME_KERNEL
When running mysqld under
gdb, you should disable the stack trace
with --skip-stack-trace to be
able to catch segfaults within gdb.
In MySQL 4.0.14 and above you should use the
--gdb option to
mysqld. This installs an interrupt handler
for SIGINT (needed to stop
mysqld with ^C to set
breakpoints) and disable stack tracing and core file handling.
It is very hard to debug MySQL under gdb if
you do a lot of new connections the whole time as
gdb doesn't free the memory for old
threads. You can avoid this problem by starting
mysqld with
thread_cache_size set to a
value equal to
max_connections + 1. In most
cases just using
--thread_cache_size=5' helps a
lot!
If you want to get a core dump on Linux if
mysqld dies with a SIGSEGV signal, you can
start mysqld with the
--core-file option. This core
file can be used to make a backtrace that may help you find
out why mysqld died:
shell> gdb mysqld core
gdb> backtrace full
gdb> quit
See Section B.1.4.2, “What to Do If MySQL Keeps Crashing”.
If you are using gdb 4.17.x or above on
Linux, you should install a .gdb file,
with the following information, in your current directory:
set print sevenbit off handle SIGUSR1 nostop noprint handle SIGUSR2 nostop noprint handle SIGWAITING nostop noprint handle SIGLWP nostop noprint handle SIGPIPE nostop handle SIGALRM nostop handle SIGHUP nostop handle SIGTERM nostop noprint
If you have problems debugging threads with gdb, you should download gdb 5.x and try this instead. The new gdb version has very improved thread handling!
Here is an example how to debug mysqld:
shell> gdb /usr/local/libexec/mysqld
gdb> run
...
backtrace full # Do this when mysqld crashes
Include the above output in a bug report, which you can file using the instructions in Section 1.6, “How to Report Bugs or Problems”.
If mysqld hangs you can try to use some
system tools like strace or
/usr/proc/bin/pstack to examine where
mysqld has hung.
strace /tmp/log libexec/mysqld
If you are using the Perl DBI interface,
you can turn on debugging information by using the
trace method or by setting the
DBI_TRACE environment variable.
On some operating systems, the error log contains a stack
trace if mysqld dies unexpectedly. You can
use this to find out where (and maybe why)
mysqld died. See
Section 5.2.2, “The Error Log”. To get a stack trace, you must
not compile mysqld with the
-fomit-frame-pointer option to gcc. See
Section 22.5.1.1, “Compiling MySQL for Debugging”.
A stack trace in the error log looks something like this:
mysqld got signal 11; Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x41fd0110 thread_stack 0x40000 mysqld(my_print_stacktrace+0x32)[0x9da402] mysqld(handle_segfault+0x28a)[0x6648e9] /lib/libpthread.so.0[0x7f1a5af000f0] /lib/libc.so.6(strcmp+0x2)[0x7f1a5a10f0f2] mysqld(_Z21check_change_passwordP3THDPKcS2_Pcj+0x7c)[0x7412cb] mysqld(_ZN16set_var_password5checkEP3THD+0xd0)[0x688354] mysqld(_Z17sql_set_variablesP3THDP4ListI12set_var_baseE+0x68)[0x688494] mysqld(_Z21mysql_execute_commandP3THD+0x41a0)[0x67a170] mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x282)[0x67f0ad] mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbb7[0x67fdf8] mysqld(_Z10do_commandP3THD+0x24d)[0x6811b6] mysqld(handle_one_connection+0x11c)[0x66e05e]
If resolution of function names for the trace fails, the trace contains less information:
mysqld got signal 11; Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x41fd0110 thread_stack 0x40000 [0x9da402] [0x6648e9] [0x7f1a5af000f0] [0x7f1a5a10f0f2] [0x7412cb] [0x688354] [0x688494] [0x67a170] [0x67f0ad] [0x67fdf8] [0x6811b6] [0x66e05e]
In the latter case, you can use the resolve_stack_dump utility to determine where mysqld died by using the following procedure:
Copy the numbers from the stack trace to a file, for example
mysqld.stack. The numbers should not include the surrounding square brackets:0x9da402 0x6648e9 0x7f1a5af000f0 0x7f1a5a10f0f2 0x7412cb 0x688354 0x688494 0x67a170 0x67f0ad 0x67fdf8 0x6811b6 0x66e05e
Make a symbol file for the mysqld server:
shell>
nm -n libexec/mysqld > /tmp/mysqld.symIf mysqld is not linked statically, use the following command instead:
shell>
nm -D -n libexec/mysqld > /tmp/mysqld.symIf you want to decode C++ symbols, use the
--demangle, if available, to nm. If your version of nm does not have this option, you will need to use the c++filt command after the stack dump has been produced to demangle the C++ names.Execute the following command:
shell>
resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stackIf you were not able to include demangled C++ names in your symbol file, process the resolve_stack_dump output using c++filt:
shell>
resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack | c++filtThis prints out where mysqld died. If that does not help you find out why mysqld died, you should create a bug report and include the output from the preceding command with the bug report.
However, in most cases it does not help us to have just a stack trace to find the reason for the problem. To be able to locate the bug or provide a workaround, in most cases we need to know the statement that killed mysqld and preferably a test case so that we can repeat the problem! See Section 1.6, “How to Report Bugs or Problems”.
Note that before starting mysqld with the general query log enabled, you should check all your tables with myisamchk. See Chapter 5, MySQL Server Administration.
If mysqld dies or hangs, you should start mysqld with the general query log enabled. See Section 5.2.3, “The General Query Log”. When mysqld dies again, you can examine the end of the log file for the query that killed mysqld.
If you use the default general query log file, the log is
stored in the database directory as
In most cases it is the last query in the log file that killed
mysqld, but if possible you should verify
this by restarting mysqld and executing the
found query from the mysql command-line
tools. If this works, you should also test all complicated
queries that didn't complete.
host_name.log
You can also try the command
EXPLAIN on all
SELECT statements that takes a
long time to ensure that mysqld is using
indexes properly. See Section 12.3.2, “EXPLAIN Syntax”.
You can find the queries that take a long time to execute by starting mysqld with the slow query log enabled. See Section 5.2.5, “The Slow Query Log”.
If you find the text mysqld restarted in
the error log file (normally named
hostname.err) you probably have found a
query that causes mysqld to fail. If this
happens, you should check all your tables with
myisamchk (see
Chapter 5, MySQL Server Administration), and test the queries
in the MySQL log files to see whether one fails. If you find
such a query, try first upgrading to the newest MySQL version.
If this doesn't help and you can't find anything in the
mysql mail archive, you should report the
bug to a MySQL mailing list. The mailing lists are described
at http://lists.mysql.com/, which also has
links to online list archives.
If you have started mysqld with
--myisam-recover, MySQL
automatically checks and tries to repair
MyISAM tables if they are marked as 'not
closed properly' or 'crashed'. If this happens, MySQL writes
an entry in the hostname.err file
'Warning: Checking table ...' which is
followed by Warning: Repairing table if the
table needs to be repaired. If you get a lot of these errors,
without mysqld having died unexpectedly
just before, then something is wrong and needs to be
investigated further. See Section 5.1.2, “Server Command Options”.
It is not a good sign if mysqld did die
unexpectedly, but in this case, you should not investigate the
Checking table... messages, but instead try
to find out why mysqld died.
If you get corrupted tables or if mysqld always fails after some update commands, you can test whether this bug is reproducible by doing the following:
Take down the MySQL daemon (with mysqladmin shutdown).
Make a backup of the tables (to guard against the very unlikely case that the repair does something bad).
Check all tables with myisamchk -s database/*.MYI. Repair any wrong tables with myisamchk -r database/
table.MYI.Make a second backup of the tables.
Remove (or move away) any old log files from the MySQL data directory if you need more space.
Start mysqld with the binary log enabled. If you want to find a query that crashes mysqld, you should start the server with both the general query log enabled as well. See Section 5.2.3, “The General Query Log”, and Section 5.2.4, “The Binary Log”.
When you have gotten a crashed table, stop the
mysqld server.Restore the backup.
Restart the mysqld server without the binary log enabled.
Re-execute the commands with mysqlbinlog binary-log-file | mysql. The binary log is saved in the MySQL database directory with the name
hostname-bin..NNNNNNIf the tables are corrupted again or you can get mysqld to die with the above command, you have found reproducible bug that should be easy to fix! FTP the tables and the binary log to ftp://ftp.mysql.com/pub/mysql/upload/ and report it in our bugs database using the instructions given in Section 1.6, “How to Report Bugs or Problems”. (Please note that the
/pub/mysql/upload/FTP directory is not listable, so you'll not see what you've uploaded in your FTP client.) If you are a support customer, you can use the MySQL Customer Support Center https://support.mysql.com/ to alert the MySQL team about the problem and have it fixed as soon as possible.
You can also use the script mysql_find_rows to just execute some of the update statements if you want to narrow down the problem.
To be able to debug a MySQL client with the integrated debug
package, you should configure MySQL with
--with-debug or
--with-debug=full. See
Section 2.10.2, “Typical configure Options”.
Before running a client, you should set the
MYSQL_DEBUG environment variable:
shell>MYSQL_DEBUG=d:t:O,/tmp/client.traceshell>export MYSQL_DEBUG
This causes clients to generate a trace file in
/tmp/client.trace.
If you have problems with your own client code, you should attempt to connect to the server and run your query using a client that is known to work. Do this by running mysql in debugging mode (assuming that you have compiled MySQL with debugging on):
shell> mysql --debug=d:t:O,/tmp/client.trace
This provides useful information in case you mail a bug report. See Section 1.6, “How to Report Bugs or Problems”.
If your client crashes at some 'legal' looking code, you should
check that your mysql.h include file
matches your MySQL library file. A very common mistake is to use
an old mysql.h file from an old MySQL
installation with new MySQL library.
The MySQL server and most MySQL clients are compiled with the DBUG package originally created by Fred Fish. When you have configured MySQL for debugging, this package makes it possible to get a trace file of what the program is debugging. See Section 22.5.1.2, “Creating Trace Files”.
This section summaries the argument values that you can specify
in debug options on the command line for MySQL programs that
have been built with debugging support. For more information
about programming with the DBUG package, see the DBUG manual in
the dbug directory of MySQL source
distributions. It is best to use a recent distribution to get
the most updated DBUG manual.
You use the debug package by invoking a program with the
--debug="..." or the -#...
option.
Most MySQL programs have a default debug string that is used if
you don't specify an option to --debug. The
default trace file is usually
/tmp/program_name.trace on Unix and
\program_name.trace on Windows.
The debug control string is a sequence of colon-separated fields as follows:
<field_1>:<field_2>:...:<field_N>
Each field consists of a mandatory flag character followed by an
optional “,” and comma-separated
list of modifiers:
flag[,modifier,modifier,...,modifier]
The following table shows the currently recognized flag characters.
| Flag | Description |
d | Enable output from DBUG_<N> macros for the current state. May be followed by a list of keywords which selects output only for the DBUG macros with that keyword. An empty list of keywords implies output for all macros. |
D | Delay after each debugger output line. The argument is the number of
tenths of seconds to delay, subject to machine
capabilities. For example, -#D,20
specifies a delay of two seconds. |
f | Limit debugging, tracing, and profiling to the list of named functions.
Note that a null list disables all functions. The
appropriate d or t
flags must still be given; this flag only limits their
actions if they are enabled. |
F | Identify the source file name for each line of debug or trace output. |
i | Identify the process with the PID or thread ID for each line of debug or trace output. |
g | Enable profiling. Create a file called dbugmon.out
containing information that can be used to profile the
program. May be followed by a list of keywords that
select profiling only for the functions in that list. A
null list implies that all functions are considered. |
L | Identify the source file line number for each line of debug or trace output. |
n | Print the current function nesting depth for each line of debug or trace output. |
N | Number each line of debug output. |
o | Redirect the debugger output stream to the specified file. The default
output is stderr. |
O | Like o, but the file is really flushed between each
write. When needed, the file is closed and reopened
between each write. |
p | Limit debugger actions to specified processes. A process must be
identified with the DBUG_PROCESS
macro and match one in the list for debugger actions to
occur. |
P | Print the current process name for each line of debug or trace output. |
r | When pushing a new state, do not inherit the previous state's function nesting level. Useful when the output is to start at the left margin. |
S | Do function _sanity(_file_,_line_) at each debugged
function until _sanity() returns
something that differs from 0. (Mostly used with
safemalloc to find memory leaks) |
t | Enable function call/exit trace lines. May be followed by a list (containing only one modifier) giving a numeric maximum trace level, beyond which no output occurs for either debugging or tracing macros. The default is a compile time option. |
Some examples of debug control strings that might appear on a
shell command line (the -# is typically used to
introduce a control string to an application program) are:
-#d:t -#d:f,main,subr1:F:L:t,20 -#d,input,output,files:n -#d:t:i:O,\\mysqld.trace
In MySQL, common tags to print (with the d
option) are enter, exit,
error, warning,
info, and loop.
I have tried to use the RTS thread packages with MySQL but stumbled on the following problems:
They use old versions of many POSIX calls and it is very tedious to make wrappers for all functions. I am inclined to think that it would be easier to change the thread libraries to the newest POSIX specification.
Some wrappers are currently written. See
mysys/my_pthread.c for more info.
At least the following should be changed:
pthread_get_specific should use one argument.
sigwait should take two arguments. A lot of
functions (at least pthread_cond_wait,
pthread_cond_timedwait()) should return the
error code on error. Now they return -1 and set
errno.
Another problem is that user-level threads use the
ALRM signal and this aborts a lot of
functions (read, write,
open...). MySQL should do a retry on
interrupt on all of these but it is not that easy to verify it.
The biggest unsolved problem is the following:
To get thread-level alarms I changed
mysys/thr_alarm.c to wait between alarms
with pthread_cond_timedwait(), but this
aborts with error EINTR. I tried to debug the
thread library as to why this happens, but couldn't find any
easy solution.
If someone wants to try MySQL with RTS threads I suggest the following:
Change functions MySQL uses from the thread library to POSIX. This shouldn't take that long.
Compile all libraries with the
-DHAVE_rts_threads.Compile
thr_alarm.If there are some small differences in the implementation, they may be fixed by changing
my_pthread.handmy_pthread.c.Run
thr_alarm. If it runs without any “warning,” “error,” or aborted messages, you are on the right track. Here is a successful run on Solaris:Main thread: 1 Thread 0 (5) started Thread: 5 Waiting process_alarm Thread 1 (6) started Thread: 6 Waiting process_alarm process_alarm thread_alarm Thread: 6 Slept for 1 (1) sec Thread: 6 Waiting process_alarm process_alarm thread_alarm Thread: 6 Slept for 2 (2) sec Thread: 6 Simulation of no alarm needed Thread: 6 Slept for 0 (3) sec Thread: 6 Waiting process_alarm process_alarm thread_alarm Thread: 6 Slept for 4 (4) sec Thread: 6 Waiting process_alarm thread_alarm Thread: 5 Slept for 10 (10) sec Thread: 5 Waiting process_alarm process_alarm thread_alarm Thread: 6 Slept for 5 (5) sec Thread: 6 Waiting process_alarm process_alarm ... thread_alarm Thread: 5 Slept for 0 (1) sec end
MySQL is very dependent on the thread package used. So when choosing a good platform for MySQL, the thread package is very important.
There are at least three types of thread packages:
User threads in a single process. Thread switching is managed with alarms and the threads library manages all nonthread-safe functions with locks. Read, write and select operations are usually managed with a thread-specific select that switches to another thread if the running threads have to wait for data. If the user thread packages are integrated in the standard libs (FreeBSD and BSDI threads) the thread package requires less overhead than thread packages that have to map all unsafe calls (MIT-pthreads, FSU Pthreads and RTS threads). In some environments (for example, SCO), all system calls are thread-safe so the mapping can be done very easily (FSU Pthreads on SCO). Downside: All mapped calls take a little time and it is quite tricky to be able to handle all situations. There are usually also some system calls that are not handled by the thread package (like MIT-pthreads and sockets). Thread scheduling isn't always optimal.
User threads in separate processes. Thread switching is done by the kernel and all data are shared between threads. The thread package manages the standard thread calls to allow sharing data between threads. LinuxThreads is using this method. Downside: Lots of processes. Thread creating is slow. If one thread dies the rest are usually left hanging and you must kill them all before restarting. Thread switching is somewhat expensive.
Kernel threads. Thread switching is handled by the thread library or the kernel and is very fast. Everything is done in one process, but on some systems, ps may show the different threads. If one thread aborts, the whole process aborts. Most system calls are thread-safe and should require very little overhead. Solaris, HP-UX, AIX and OSF/1 have kernel threads.
In some systems kernel threads are managed by integrating user level threads in the system libraries. In such cases, the thread switching can only be done by the thread library and the kernel isn't really “thread aware.”