Loading...
Build a Go Elasticsearch client with Docker
Marco Franssen
8 min read•1491 words
In this post, I'll walk through the Go client released alongside Elasticsearch 7.0-rc1. We'll build a small Go API and use Docker Compose to run an Elasticsearch cluster for our example.
In my previous post, I covered some Docker tips and tricks that we'll use again here.
Initializing your project
First, we need a project folder. Inside it, we'll initialize our Go module and add a Dockerfile and docker-compose.yml. Once those are in place, we can start developing. I like to bootstrap projects with as few mouse clicks as possible, so here are a few shell commands to create our project outline.
mkdir es-demo
cd es-demo
go mod init github.com/marcofranssen/es-demo
go get -u github.com/elastic/go-elasticsearch/v7@7.x
touch Dockerfile
touch docker-compose.yml
touch main.goWe'll use the same Dockerfile definition as in my previous post. You can find its contents below.
Dockerfile
Loading es-demo/Dockerfile…
My previous post explains every line of this Dockerfile. Have a look there if you'd like to understand the details.
Docker Compose
Next, let's add the contents of docker-compose.yml to define our runtime infrastructure and container setup.
Loading es-demo/docker-compose.yml…
As you can see, we expose only the web container's port to our host, so the other services aren't directly accessible from the host. We can access our API on port 5000.
Webserver boilerplate
Finally, I'll use the graceful web server I wrote about earlier as the starting point for our Elasticsearch-backed API.
Loading graceful-webserver/main.go…
With all of this in place, we can start our infrastructure and take a first look at the setup in our browser.
NOTE: We haven't used the Go Elasticsearch client yet. We'll get to that once we've confirmed the infrastructure is working.
Run our solution with docker-compose up --build. This builds our API image and downloads the Elasticsearch Docker images. At http://localhost:5000, you should see a blank response with HTTP status 200. You'll also notice that http://localhost:9200 isn't available, because we haven't exposed that port from our Docker network.
Time for code
Now we're ready to update our web server to interact with Elasticsearch. We'll use the go-elasticsearch client we installed in the initial setup. Let's add an import statement and a function to create an Elasticsearch client.
import (
"github.com/elastic/go-elasticsearch/v7"
)… omitted for brevity …
func newEsClient(logger *log.Logger, addresses []string) *elasticsearch.Client {
cfg := elasticsearch.Config{Addresses: addresses}
client, err := elasticsearch.NewClient(cfg)
if err != nil {
logger.Println(err)
panic(err)
}
return client
}Now we can create an Elasticsearch client in our main function and use it in our HTTP server. Let's add another command-line flag to configure the Elasticsearch addresses and create the client from them.
var (
addr string
esAddresses string
)
func main() {
flag.StringVar(&esAddresses, "es-addresses", "http://es01:9200,http://es02:9200", "elasticsearch addresses")
es := newEsClient(logger, strings.Split(esAddresses, ","))
}Next, let's update our web server function to return Elasticsearch cluster information when we call the endpoint. We'll pass the client to the web server function and update the HTTP handler registered at / to fetch that information.
func newWebserver(logger *log.Logger, es *elasticsearch.Client) *http.Server {
router := http.NewServeMux()
router.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
logger.Println(r.Method, r.URL.Path, r.RemoteAddr, r.UserAgent())
read, write := io.Pipe()
go func() {
defer write.Close()
esInfo, err := es.Info()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
} else {
defer esInfo.Body.Close()
io.Copy(write, esInfo.Body)
}
}()
io.Copy(w, read)
w.WriteHeader(http.StatusOK)
})
return &http.Server{
Addr: listenAddr,
Handler: router,
ErrorLog: logger,
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 15 * time.Second,
}
}In this example, we're piping the Elasticsearch response body to our HTTP response writer using io.Pipe() and io.Copy(). This lets you add your own logic, such as authentication, while streaming the Elasticsearch response directly to your API response. It's memory-efficient because we don't load the entire response into memory before sending it.
All that's left is to update the call to newWebserver so we pass it the Elasticsearch client.
server := newWebserver(logger, esClient)Let's try the example with docker-compose up --build -d. This rebuilds the web container and runs it in the background, making it easier to continue with the next example. To view the logs, use docker-compose logs -f, or docker-compose logs -f web to show only the web container's logs.
Searching an index
For a more concrete example, let's search an index as a starting point for your Elasticsearch Go journey.
First, we'll define a new HTTP handler on our router to forward our search request to Elasticsearch. I'll also add a helper function to build the query and a string constant for the fixed part of it.
const searchMatch = `
"query": {
"multi_match": {
"query": %q,
"fields": ["lastName^100", "firstName^10", "country", "title"],
"operator": "and"
}
},
"highlight": {
"fields": {
"lastName": { "number_of_fragments": 0 },
"firstName": { "number_of_fragments": 0 },
"country": { "number_of_fragments": 0 },
"title": { "number_of_fragments": 0 }
}
},
"size": 25,
"sort": [{ "_score": "desc" }, { "_doc": "asc" }]`
func buildQuery(query string) io.Reader {
var b strings.Builder
b.WriteString("{\n")
b.WriteString(fmt.Sprintf(searchMatch, query))
b.WriteString("\n}")
return strings.NewReader(b.String())
}
func newWebserver(logger *log.Logger, es *elasticsearch.Client) *http.Server {
router := http.NewServeMux()
router.HandleFunc("/search", func(w http.ResponseWriter, r *http.Request) {
logger.Println(r.Method, r.URL.Path, r.RemoteAddr, r.UserAgent())
q := r.URL.Query().Get("q")
read, write := io.Pipe()
go func() {
defer write.Close()
res, err := es.Search(
es.Search.WithContext(r.Context()),
es.Search.WithIndex("people"),
es.Search.WithBody(buildQuery(q)),
es.Search.WithTrackTotalHits(true),
)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
} else {
defer res.Body.Close()
io.Copy(write, res.Body)
}
}()
io.Copy(w, read)
})
return &http.Server{
Addr: listenAddr,
Handler: router,
ErrorLog: logger,
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 15 * time.Second,
}
}This code adds a handler for /search. It uses the query string parameter q to build an Elasticsearch search request, with the query sent as a JSON body in a POST request. Let's try it with docker-compose up --build -d. You'll get an error response when calling the API because we don't have an index called people yet. Try it at http://localhost:5000/search?q=marco.
To resolve this, we'll add some code to create the index and populate Elasticsearch with a few initial records.
import (
"github.com/elastic/go-elasticsearch/v7/esapi"
)
func bootstrap(es *elasticsearch.Client) error {
idx := "people"
ctx := context.Background()
_, err := esapi.IndicesDeleteRequest{Index: []string{idx}}.Do(ctx, es)
if err != nil {
return err
}
_, err2 := esapi.IndicesCreateRequest{Index: idx}.Do(ctx, es)
if err2 != nil {
return err2
}
people := make([]*Person, 0, 4)
people = append(people, &Person{
ID: "1",
Title: "Mr.",
FirstName: "Marco",
LastName: "Franssen",
Email: "marco.franssen@elasticsearch.com",
Country: "The Netherlands",
})
people = append(people, &Person{
ID: "2",
Title: "Mr.",
FirstName: "John",
LastName: "Doe",
Email: "john.doe@elasticsearch.com",
Country: "Neverland",
})
people = append(people, &Person{
ID: "3",
Title: "Mrs.",
FirstName: "Jane",
LastName: "Doe",
Email: "jane.doe@golang.org",
Country: "Neverland",
})
people = append(people, &Person{
ID: "4",
Title: "Mr.",
FirstName: "Rob",
LastName: "Pike",
Email: "rob.pike@golang.org",
Country: "Unknown",
})
for _, p := range people {
payload, err := json.Marshal(p)
if err != nil {
return err
}
_, err3 := esapi.CreateRequest{
Index: idx,
DocumentID: p.ID,
Body: bytes.NewReader(payload),
}.Do(ctx, es)
if err != nil {
return err3
}
}
return nil
}Finally, call this bootstrap function in your main function, right after creating the Elasticsearch client.
err := bootstrap(esClient)
if err != nil {
panic(err)
}Now we can run the solution again and try some searches. Use docker-compose up -d --build, then docker-compose logs -f to see requests reach your handler. As you might have noticed, we've enabled searches only by first name, last name, country, and title. We've also weighted those fields to influence the score when a query matches one or more of them. Here are some queries to try:
- http://localhost:5000/search?q=mr
- http://localhost:5000/search?q=marco
- http://localhost:5000/search?q=Rob
- http://localhost:5000/search?q=Neverland
- http://localhost:5000/search?q=elasticsearch
The last search won't return any results because we don't search email addresses. Try editing the query template to include the email field, then rerun your code with docker-compose up -d --build and give that search another go.
Resources
Here are some resources to explore if you'd like to learn more.
- Go Docker tips and tricks
- Go web server with graceful shutdown
- Elasticsearch Docker image
- Go Elasticsearch GitHub project
Thanks for reading! I hope this gives you a useful starting point for your own Go Elasticsearch projects. Please share it with your friends and colleagues on social media. Oh, and before I forget, you can download the complete solution here as a zip.
