This chapter covers how full text searches work in Cyclos, and how to fine-tune them. Full text searches allows retrieving documents using its words, returning documents that match a given textual query (often related as keywords in Cyclos). The full text engine processes words both when indexing (calculating the words on documents) and querying (transforming an input text in a way it matches indexed documents):
Currently the following data types are searched with full text queries when using keywords:
Cyclos uses the native PostgreSQL's full text indexing capabilities. However, as the native query syntax can be too much formal for end users, a query preprocessor is included in Cyclos, such that the following variants are supported:
PostgreSQL has two main data types related to full text searches: TSVECTOR which represent an indexed text and TSQUERY which represents a textual query. In order to process either data type, a dictionary is used. The concept of a dictionary is very important, as each dictionary knows how to perform the processing (like stemming and stopwords removal). The same dictionaries must be used used when processing TSVECTOR and TSQUERY, or queries will probably never match, even when searching a correct term.
Cyclos controls the dictionaries to be used on a specific network on the network (or global) default configuration. From there it is possible to set which dictionaries are used for all data (except translation keys, which are per language, and hence, uses a dictionary for that language). Whenever the dictionaries are changed, all data in the network is indexed on the background, so be careful on large databases, as this process may take some time. Make sure to select the dictionaries for all languages the network uses. Also, be aware that selecting more dictionaries will probably cause the TSVECTOR columns to be larger, and may impact the performance on queries. To prevent this, is possible to use only the default dictionary (which is a simple dictionary that just uses the PostgreSQL's unaccent module). The default dictionary doesn't benefit from language-specific processing (like stemming or stopwords), but will work for any language.
If needed, it is possible to create or modify the PostgreSQL dictionaries to match your needs. For example, dictionaries for some important languages (like Japanese or Chinese) are not included by default in PostgreSQL. Please, consult the PostgreSQL documentation on dictionaries on how to define a new dictionary. Then, to make Cyclos use that dictionary, insert a record in the dictionaries table, which has 2 columns: dictionary (the name of the created dictionary in PostgreSQL) and name (the display name shown in Cyclos). Finally, as usual, set the default configuration to use that dictionary.