Elasticsearch

Elasticsearch provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. It not only great for search, but also for extracting value out of all the maps that businesses are collecting.

In Ramani Cloud we’ve made an API for 1) getting maps from PostGIS to Elasticsearch in a very simple way and 2) exposing the Elasticsearch search API in a secure manner.

Getting maps from PostGIS to Elasticsearch

It's easy to get maps from PostGIS to Elasticsearch. It's actual just a click on a button. But first you can setup the mapping between PostGIS fields types and Elasticsearch fields types.

  1. Select which table (or other kind of relation) you want to index in Elasticsearch
  2. Click the Elasticsearch tab in in lower panel.
  3. The Native PG type column shows the field types in PostGIS. The Elasticsearch type column shows the suggested types in Elasticsearch. The latter you can change by double clicking on a value.
  4. The columns Index to Format sets how the values should be indexed, analysed, formatted etc. Look further down for more information about setting up custom analyzers.
  5. Click the (Re)index in Elasticsearch to index the data. If the index exist it will be deleted and indexed again.
  6. You can delete an Index by clicking Delete from Elasticsearch.

After indexing, you can search the index on:

https://host.com/api/v2/elasticsearch/search/[database]/[schema]/[table]

You can see the mapping document by calling this:

https://host.com/api/v2/elasticsearch/map/[database]/[schema]/[table]?key=[your API key]

The actual index in Elasticsearch will be named: database_schema_table

Document structure in Elasticsearch

The indexed documents are formatted as GeoJSON. And example of GeoJSON:

{
  "type": "Feature",
  "geometry": {
    "type": "Point",
    "coordinates": [125.6, 10.1]
  },
  "properties": {
    "name": "Dinagat Islands"
  }
}

Exposing the Elasticsearch search API

Ramani Cloud exposes only the search API of Elasticsearch. You can use the Ramani Cloud API as you would use the search API of Elasticsearch.

Below is an example of a GET with a URI search. The following examples use the cURL program, but any HTTP client can be used.

curl -i --header "Content-Type: application/json" -X GET \
"https://gc2.io/api/v2/elasticsearch/search/dk/matrikel/jordstykke_view?q=properties.string1:Engelsholm&size=1&pretty"

And here with a request body. The body can be sent as either GET and POST. The latter can be used in clients who can not GET with body. For example, web browsers: 

curl -i --header "Content-Type: application/json" -X GET \
https://gc2.io/api/v2/elasticsearch/search/dk/matrikel/jordstykke_view --maps \
'{
    "query": {
        "match" : {
            "properties.string1" : "Engelsholm"
        }
    }
}'

Set up Elasticsearch index settings and add custom analyzers

If you host your own Ramani Cloud, you can setup Elasticsearch index settings including custom analyzers and search analyzers. The analyzers can then be selected in the Ramani Cloud Admin.

On the Ramani Cloud server create an document called app/conf/elasticsearch_settings.json and fill in the index settings:

{
  "settings": {
    "number_of_shards": 4,
    "analysis": {
      "analyzer": {
        "str_search_analyzer": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": ["lowercase"]
        },
        "str_index_analyzer": {
          "type": "custom",
          "tokenizer": "whitespace",
          "filter": [
            "lowercase",
            "substring"
          ]
        }
      },
      "filter": {
        "substring": {
          "type": "edgeNGram",
          "min_gram": 1,
          "max_gram": 255
        }
      }
    }
  }
}

All new indexes will be created with these settings.