With pgvector, vector storage and similarity searching become standard.

Execute CREATE EXTENSION on the database to use this feature. Then use the psql command to connect to the database to be set up.

postgres=# CREATE EXTENSION vector;
CREATE EXTENSION

When using the automatic password generation function of SUPERUSER andsetting up this feature, refer to "6.7.1 CREATE EXTENSION" and execute CREATE EXTENSION.

Using pgvectorscale can speed up processing of vector data.

Execute CREATE EXTENSION on the database to use this feature. Adding the CASCADE option will also CREATE EXTENSION any dependent pgvector. Then use the psql command to connect to the database to be set up.

postgres=# CREATE EXTENSION IF NOT EXISTS vectorscale CASCADE;
CREATE EXTENSION

When using the automatic password generation function of SUPERUSER and setting up this feature, refer to "6.7.1 CREATE EXTENSION" and execute CREATE EXTENSION.

This feature only supports x86.

Vector databases speed up responses with approximate searches. The comparison between approximate and full search can be represented by a metric called recall.

By measuring recall, you can measure the accuracy of vector database searches.

By regularly measuring the recall rate, you can ensure that the target recall rate is met. If the recall falls below the target, consider tuning the vector database parameters or rebuilding the index.

Operator allows you to periodically measure and monitor the recall of a vector database. This section describes how to enable periodic measurement and monitoring of vector database recall.

By setting spec.fep.measurement.recallForVector.enable to true for the FEPCluster custom resource, the vector database recall measurement is enabled.

When recall measurement is enabled for a vector database, a measurement Cronjob is built. The measurement Cronjob periodically performs approximate and full searches on random vectors for objects in a specified vector database to measure the recall rate.

The measured results are averaged and stored in the table vector_database_recall_summary. By looking at the table, you can see the average recall at the time of measurement.

When the vector database reproduction rate measurement is enabled, by setting spec.fep.monitoring.enable to true and enabling the monitoring feature of FEPExporter, you can collect the information of table vector_database_recall_summary as a metric and monitor the vector database recall rate.

In addition to enabling monitoring, you can define parameter spec.fep.measurement.recallForVector.alertThreshold for the FEPCluster custom resource so that the AlertManager can alert you when the vector database recall falls below the specified value.

Provide an example of defining a custom FEPCluster resource that enables vector database recall measurement.

In this example definition, the measurement starts at 0:00 on the first day of each month and ends 40 hours after the measurement starts.

spec:
  fep:
    measurement:
      recallForVector:
        enable: true
        schedule: "0 0 1 * *"
        maxDuration: "40h"
        parallelJobs: 5
        topK: 50
        alertThreshold: 0.7 #Specify the alert threshold when monitoring is enabled

        #Specify the database object to measure
        targets:
        - database: "mydb"
          tableConfigs: #Specify multiple objects for the database
          - schemaObject: "schema.table1.column1"
          - schemaObject: "schema.table2.column2"
            distanceMetric: l2

    #Enabling vector database recall measurement and monitoring enables recall monitoring in Prometheus 
    monitoring:
      enable: true

Specify a schedule for vector database recall measurement at a time when the workload is low and there is little impact from the measurement.

The table vector_database_recall_summar which stores the result of the recall measurement of the vector database, contains the following information. A row is inserted for each object being measured and the information is updated as it is measured.

If you enable the vector database recall measurement and set spec.fep.monitoring.enable to true for the FEPCluster custom resource, the FEPExporter monitoring feature enables recall monitoring.

Metrics that are valid when vector database recall measurement is enabled

You can also add the following alert rule by defining the FEPCluster custom resource parameter spec.fep. measurement.recallForVector.alertThreshold:

Alert Rule Added