Amazon ElasticSearch Java Integration Testing

Amazon offers ElasticSearch Cloud service. As you may know, Amazon is based on a fork called Opendistro after some issues with the original product. This Amazon product has its own lifecycle with versioning tags different from the original ElasticSearch product. So when it comes to test your Java client code against server to check integration, it’s not easy to come up with a valid environment that closely simulates what is happening with the actual Amazon product.

So, this article is offering a working example on how to do integration testing for your Java clients as part of your project build, and detect integration issues before even going to the Amazon environment. This will allow your development team to identify issues very soon in your process, and be capable of reproducing production issues in the dev env for quick fixing.

Dependencies

Let’s start with the dependencies list:

We are using JUnit5 here, but we could use any other version, or testing framework like TestNG, is not critical for this setup.

Then we need Testcontainers, which is the basic testing tool required here to work with Docker images, and being able to bootstrap the Amazon ElasticSearch server for testing. For futher Testcontainers Requirements check link.

The Java ElastiSearch Client dependency which will be used to send queries to the server.

Test Skeleton

As you can see we are using the Amazon Docker image. In our case we use version 1.11.0, which seems to match the one we get on Amazon Cloud with 7.9 version. Amazon provides this FAQ , but it’s not fully comprehensive on versioning in my opinion. To check version matching just send a HTTP GET to the Elasticsearch root server and you will get something like this:

curl -XGET https://172.17.0.3:9200 -u admin:admin — insecure
{
“name” : “4c3f689ddd76”,
“cluster_name” : “docker-cluster”,
“cluster_uuid” : “n96mfPN3QYOx3kPb2NdcWg”,
“version” : {
“number” : “7.9.1”,
“build_flavor” : “oss”,
“build_type” : “tar”,
“build_hash” : “083627f112ba94dffc1232e8b42b73492789ef91”,
“build_date” : “2020–09–01T21:22:21.964974Z”,
“build_snapshot” : false,
“lucene_version” : “8.6.2”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

Properties worth mentioning are:

  • Number: This is the Amazon product version. It should correlate with the one you set in Amazon Console for your ElasticSearch domain.
  • Lucene_version : This is the version of internal Lucene search engine used by the server.
  • Minimum wire Compatibility version: Minimum version for other nodes in the cluster to coordinate properly.
  • Minimum Index Compatibility : Same as previous but related to reading files from an older node.

During “prepareESContainer” BeforeAll method we prepare the container and start it. As explained in docker readme, we expose ports 9200 and 9600. We also set environment variable to have single node server:

Notice we added a condition to wait for a log message to be printed by server. This message is printed when API interface is ready to receive requests.

Then we go on by creating the Java clients. You will notice there is some security tweaks to allow accessing the local server. If that is critical for your testing, you will need to figure out how to access the local instance using the proper localhost defined in the server certificate, so they match properly:

The presented test is as simple as sending a request to the Health resource:

Of course after testing we just cleanup all the resources as usual:

You are basically ready to add your own tests so go ahead and enjoy, cheers…

Index Management and Data population

Your testsuite may involve index management, or just queries, or maybe both. In our case we wanted to check our Java Lambda code queries, but this requires to have actual indexes and data in the server to exercise the queries. I’m sharing some tools we use to create indexes, templates, and load some data.

We invoke “prepareIndexes” from the BeforeAll setup method. This allow us to create index templates, indexes and then load some actual data:

The actual index templates and data files containing documents are located in the test resources directory, and we load them from the classloader.

Conclusion

This small test environment should allow you to test any interaction with the server, and check you client code is properly behaving. We hope this article allows other teams to improve their process by adding integration test to their Java Amazon ElasticSearch clients.

This solution was a combined effort from the Core Network team in Telestax, and hence the achievement comes from the team, kudos for the team!!

Software Engineer with more than 15 years experience in the Telco sector. Currently working at Telestax.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store