k-NN index

The k-NN plugin introduces a custom data type, the knn_vector, that allows users to ingest their k-NN vectors into an OpenSearch index and perform different kinds of k-NN search. The knn_vector field is highly configurable and can serve many different k-NN workloads. For more information, see k-NN vector.

Lucene byte vector

Starting with k-NN plugin version 2.9, you can use byte vectors with the lucene engine in order to reduce the amount of storage space needed. For more information, see Lucene byte vector.

Method definitions

A method definition refers to the underlying configuration of the Approximate k-NN algorithm you want to use. Method definitions are used to either create a knn_vector field (when the method does not require training) or create a model during training that can then be used to create a knn_vector field.

A method definition will always contain the name of the method, the space_type the method is built for, the engine (the library) to use, and a map of parameters.

Supported nmslib methods

HNSW parameters

For nmslib, ef_search is set in the index settings.

An index created in OpenSearch version 2.11 or earlier will still use the old ef_construction value (512).

Supported faiss methods

For hnsw, “innerproduct” is not available when PQ is used.

HNSW parameters

An index created in OpenSearch version 2.11 or earlier will still use the old ef_construction and ef_search values (512).

IVF parameters

For more information about setting these parameters, refer to the Faiss documentation.

IVF training requirements

The IVF algorithm requires a training step. To create an index that uses IVF, you need to train a model with the Train API, passing the IVF method definition. IVF requires that, at a minimum, there should be nlist training data points, but it is recommended that you use more. Training data can be composed of either the same data that is going to be ingested or a separate dataset.

Supported Lucene methods

HNSW parameters

Lucene HNSW implementation ignores ef_search and dynamically sets it to the value of “k” in the search request. Therefore, there is no need to make settings for ef_search when using the Lucene engine.

An index created in OpenSearch version 2.11 or earlier will still use the old ef_construction value (512).

{
    "type": "knn_vector",
    "dimension": 100,
    "method": {
        "name":"hnsw",
        "engine":"lucene",
        "space_type": "l2",
        "parameters":{
            "m":2048,
            "ef_construction": 245
        }
    }
}

Supported faiss encoders

You can use encoders to reduce the memory footprint of a k-NN index at the expense of search accuracy. faiss has several encoder types, but the plugin currently only supports flat and pq encoding.

The following example method definition specifies the hnsw method and a pq encoder:

"method": {
  "name":"hnsw",
  "engine":"faiss",
  "space_type": "l2",
  "parameters":{
    "encoder":{
      "name":"pq",
      "parameters":{
        "code_size": 8,
        "m": 8
      }
    }
  }
}

The hnsw method supports the pq encoder for OpenSearch versions 2.10 and later. The code_size parameter of a pq encoder with the hnsw method must be 8.

Examples

The following example uses the ivf method without specifying an encoder (by default, OpenSearch uses the flat encoder):

"method": {
  "name":"ivf",
  "engine":"faiss",
  "space_type": "l2",
  "parameters":{
    "nlist": 4,
    "nprobes": 2
  }
}

The following example uses the ivf method with a pq encoder:

"method": {
  "name":"ivf",
  "engine":"faiss",
  "space_type": "l2",
  "parameters":{
    "encoder":{
      "name":"pq",
      "parameters":{
        "code_size": 8,
        "m": 8
      }
    }
  }
}

The following example uses the hnsw method without specifying an encoder (by default, OpenSearch uses the flat encoder):

"method": {
  "name":"hnsw",
  "engine":"faiss",
  "space_type": "l2",
  "parameters":{
    "ef_construction": 256,
    "m": 8
  }
}

PQ parameters

Choosing the right method

There are a lot of options to choose from when building your knn_vector field. To determine the correct methods and parameters to choose, you should first understand what requirements you have for your workload and what trade-offs you are willing to make. Factors to consider are (1) query latency, (2) query quality, (3) memory limits, (4) indexing latency.

If memory is not a concern, HNSW offers a very strong query latency/query quality tradeoff.

If you want to use less memory and index faster than HNSW, while maintaining similar query quality, you should evaluate IVF.

If memory is a concern, consider adding a PQ encoder to your HNSW or IVF index. Because PQ is a lossy encoding, query quality will drop.

Memory estimation

In a typical OpenSearch cluster, a certain portion of RAM is set aside for the JVM heap. The k-NN plugin allocates native library indexes to a portion of the remaining RAM. This portion’s size is determined by the circuit_breaker_limit cluster setting. By default, the limit is set at 50%.

Having a replica doubles the total number of vectors.

HNSW memory estimation

The memory required for HNSW is estimated to be 1.1 * (4 * dimension + 8 * M) bytes/vector.

As an example, assume you have a million vectors with a dimension of 256 and M of 16. The memory requirement can be estimated as follows:

1.1 * (4 * 256 + 8 * 16) * 1,000,000 ~= 1.267 GB

IVF memory estimation

The memory required for IVF is estimated to be 1.1 * (((4 * dimension) * num_vectors) + (4 * nlist * d)) bytes.

As an example, assume you have a million vectors with a dimension of 256 and nlist of 128. The memory requirement can be estimated as follows:

1.1 * (((4 * 256) * 1,000,000) + (4 * 128 * 256))  ~= 1.126 GB

Index settings

Additionally, the k-NN plugin introduces several index settings that can be used to configure the k-NN structure as well.

At the moment, several parameters defined in the settings are in the deprecation process. Those parameters should be set in the mapping instead of the index settings. Parameters set in the mapping will override the parameters set in the index settings. Setting the parameters in the mapping allows an index to have multiple knn_vector fields with different parameters.

An index created in OpenSearch version 2.11 or earlier will still use the old ef_construction and ef_search values (512).