This chapter covers how full text searches work in Cyclos, and how to fine-tune them. Full text searches allows retrieving documents using its words, returning documents that match a given textual query (often related as keywords in Cyclos). The full text engine processes words both when indexing (calculating the words on documents) and querying (transforming an input text in a way it matches indexed documents). Some examples of such processing include:
Currently the following data types are searched with full text queries when using keywords:
By default, Cyclos uses the native PostgreSQL's full text indexing capabilities. Also, for geo-spatial distance filters, Cyclos uses PostGIS. But it is also possible to use an external Elasticsearch server to perform these searches. For details on the Elasticsearch integration see the Section 2.1, “Using Elasticsearch” section.
As the PostgreSQL native query syntax can be too much formal for end users, a query preprocessor is included in Cyclos, such that the following variants are supported:
Previously, Cyclos used distinct PostgreSQL text search configurations, called dictionaries in Cyclos, that defined the kind of language processing to be applied. This, however, sometimes led to confusion, as using the wrong dictionaries could actually prevent finding existing data. For example, in dutch, the phrase "van der meer" would be totally discarded, as they are all stopwords. Even though, the same phrase can appear in names, requiring the default dictionary to be selected as well.
Starting with Cyclos 4.12, the actual language processing is only performed when using Elasticsearch. In PostgreSQL, Cyclos will always use a text search configuration that doesn't remove any stopword, but removes all diacritics. So, for example 'façade' is stored as 'facade'. This should be more than ok for most systems, specially because we search by prefix by default. So, for example, seaching 'car' will match all of these: 'car', 'cars' and 'cartoon', but ranking 'car' better. For more complex language processing, use Elasticsearch.
Searching on PostgreSQL with large databases, using several custom fields and distance filter may present unacceptable performance. For such cases, or for more advanced language analysis, starting with Cyclos 4.12, it is possible to perform such searches using Elasticsearch, a well-known product which is very fast, and designed to scale horizontally. Some cloud providers, such as Amazon, even provides Elasticsearch as a service.
Important: Starting with Cyclos 4.14, the required Elasticsearch version is 7.x. Previous versions are not supported.
The Elasticsearch server / cluster needs to be deployed in a server accessible to Cyclos via its REST API. Then, in cyclos.properties, add the following properties:
Once set-up, when restarted, Cyclos will attempt to find the following Elasticsearch indexes, and, if not found, will create them and index all entities in the database: users, ads and records (translation keys are always searched in Postgresql). The indexing will be executed on the background, and it can take a few minutes, depending on the database size, to index all data.
Textual searches (referred as keywords in Cyclos) are passed to Elasticsearch using its Simple Query String syntax, which is like the one employed by Cyclos when searching on the database, but more powerful.
The actual language processing (such as removing stopwords and stemming text) is performed only on the following fields: advertisement title and description, and custom fields whose setting "Value match" is set to "Language". Note that the "Value match" field will only show up in custom fields when Elasticsearch is used. When handling searches in the database, only the checkbox "Use exact matching on search filters" will show up.
Language-analyzed fields are stored multiple times in the index: one using Elasticsearch Standard Analyzer (to prevent mismatches or when searching as users which uses another language), and another one for each language the owner of the data can have. So, for example, an advertisement owned by a user whose configuration allows English, Portuguese and French will have the field stored in all such analyzers, plus the standard.
When searching for the data, Cyclos uses all languages the logged user has, plus the standard analyzer. So, following the same previous example, if a logged user had only the French language, it would search in both the standard analyzer field, plus the French one. And if another user could see that advertisement, but have only the Spanish language, they would search in both Spanish (in which the no data exist) and in standard, and would ultimately find the advertisement using the standard analyzer.
Being a separated data store, the data on PostgreSQL database and on Elasticsearch might become de-synchonized. The database is always considered correct, and is the definitive trusted store. The data on Elasticsearch is updated automatically, as soon as the corresponding user, advertisement or record is modified on the database. But it might happen that an update request fails, or that the Elasticsearch server to be offline for some time.
Important: If you have modified a profile field (its value or the internal name) then you must reindex users, andvertisements and records.
To handle these cases, Cyclos offers methods that can be executed by scripts, directly through the menu System > Tools > Run script. Here are some examples:
Reindex ALL data on ALL indexes:
searchHandler.reindex()
Reindex ALL data on users:
userSearchHandler.reindex()
Reindex ALL data on advertisements:
adSearchHandler.reindex()
Reindex ALL data on records:
recordSearchHandler.reindex()
Other examples:
//Reindex ALL records of a given record type
import org.cyclos.entities.users.RecordType
def recordType = entityManagerHandler.find(RecordType, "recordTypeInternalName")
recordSearchHandler.reindexByType(recordType)
//Reindex a single user
import org.cyclos.entities.users.User
def user = conversionHandler.convert(User, 'loginName')
userSearchHandler.index(user)
//Reindex a single advertisement by external (masked) id (would be similar to records)
import org.cyclos.entities.marketplace.BasicAd
def ad = entityManagerHandler.find(BasicAd, unmaskId(123456789L))
adSearchHandler.index(ad)