How to import my custom XML files in ElasticSearch, luckily Logstash is there to help. Let’s create an example XML file that we want to import in Elasticsearch. Copy the text below and save this as ‘.xml’, you can also use your own XML.

<xmldata>
 <head1>
  <key1>Value1</key1>
  <key2>Value2</key2>
  <id>0001</id>
  <date>01-01-2016 09:00:00</date>
 </head1>
 <head2>
  <key3>Value3</key3>
 </head2>
</xmldata>

Building the Logstash config file

XML files can be readed by Logstash with the multi line option under the input file. Below I show my example configuration file;

input {
 file {
  path => "/config-dir/test.xml"
  start_position => beginning
  codec => multiline
  {
   pattern => "^<\?xmldata .*\>"
   negate => true
   what => "previous"
  }
 }
}

The filter part of the configuration will read the XML. This example will filter out specific values; Id, Date and Key1. Also the date value will be translated to correctly be used by Elasticsearch and Kibana.

filter {
  xml {
   store_xml => false
   source => "message"
   xpath =>
   [
    "/xmldata/head1/id/text()", "id",
    "/xmldata/head1/date/text()", "date",
    "/xmldata/head1/key1/text()", "key1"
   ]
}

date {
    match => [ "date" , "dd-MM-yyyy HH:mm:ss" ]
    timezone => "Europe/Amsterdam"
}

}

Under the output part we configure Elasticsearch and the stout so we can see the output directly in the console. In this example the Elasticsearch document_id is used from the XML file. Optional you can set the document_type part.

 
output { 
 stdout { codec => rubydebug } 
 elasticsearch { 
  index => "logstash-xml"
  hosts => ["10.0.1.3:9200"] 
  document_id => "%{[id]}"
  document_type => "xmlfiles" 
 } 
} 

Running LogStash

When you put the three pieces together, input/filter/output, you have the complete configuration file for LogStash. Save this file as logstash-xml.conf, you can test the config with the following command:

bin/logstash -f logstash-xml.conf --configtest

And run the config file with;

bin/logstash -f logstash-xml.conf