Monitor network performance with iperf3 and the Elastic Stack

By | July 31, 2016

iperf is a great tool to get network performance measurements that show the quality of the network connection between your machine and an iperf server. The iperf server can be a publicly available one or you can host it yourself. If you’re not familiar with iperf, I recommend you glance over this page.

While great to have iperf running periodically to record some network measurements, it would be great to have a centralized place to store the measurements of our gazillion servers and endpoints. Heck, even low-powered devices or smartphones could run iperf.

The Elastic Stack is an integrated set of platforms for handling data analytics. It contains:

  • Beats – a versatile, lightweight data shipper. We will use it to send out the iperf data from the monitored device. Runs on almost anything.
  • Logstash – a powerful ETL platform with custom plugin support. We use it to do some arithmetic and prepare the data for storage.
  • Elasticsearch – a resilient, distributed JSON data store with extensive aggregation and full text search support. Here our data will live.
  • Kibana – a rich visualization platform for Elasticsearch
    • Timelion – a chainable query language for visualizing time series data. Open source plugin from Elastic.
  • json-minify – an NPM-installable tool to de-beautify/minify the multiline JSON output of iperf3, basically convert a multiline, indented JSON string to a one-liner again. Unfortunately, iperf3 cannot do single-line JSON itself, and as of now Filebeat 5.0-alpha4 cannot handle multiline JSON input (although there is an issue open for it)

We’re going to use the latest 5.0.0-alpha4 version of this stack, because it offers the bleeding edge. If you need instructions on how to install the stack on your platform, you can take a look at the Elastic website at their excellent documentation. From here on I will assume that the tools are installed and running.

Our pipeline will look like this:

  1. Generate data on disk with iperf, wrapped in a bash script
  2. Send the data from disk with Filebeat to Logstash
  3. Do some arithmetic in Logstash
  4. Set up Elasticsearch
  5. Use Kibana and Timelion to visualize the performance

All artifacts related to this post are available as a repo on GitHub.

Generate data on disk with iperf3

iperf runs on Windows, Linux, Android, MacOS X, FreeBSD, OpenBSD, NetBSD, and others. For the sake of preventing duplication, I’m going to refer you to the official iperf website to get it running on your platform. And let’s immediately take the opportunity to go with iperf3 instead of 2. iperf will perform a number of network benchmarks, like: ping, jitter, packet loss, bandwidth. Things we need to determine network connection performance and health.

We’re going to use one of the publicly available iperf servers to run our tests against.

Create a wrapper script for iperf3

Depending on your OS, you might want to alter the wrapper script. This one is made for OSX and should work in Linux, too. It’s a bit too much to paste the script here, please find it in the GitHub repo.

Couple of points about this script:

  1. I’m running iperf3 two times per iteration, once for upload (default behavior) and once for download (“reverse”, -R flag).
  2. It’s important to understand that iperf3 uses the same fields but with contradicting definitions: for example, the field end.sum_received.bits_per_second during a download test means “download speed”, but during an upload test it’s describing “upload speed”. Seems perfectly logical, no? Well if you’re going to plot them on a chart, it will go nowhere. More on that later.
  3. I’m not sure how happy the maintainers of the public iperf3 servers are with inifitely long testing runs, use at your own risk 🙂

Be sure to add iperf3 to $PATH to make this work.

This script will run iperf3 for us to a server of your choosing. You might want to run it a finite number of times during dev, and keep it running in production. The script will help you out either way. Now that that’s done, it’s time to look at Filebeat to get this data off the benchmarked machine and into the pipeline.

Send the data from disk with Filebeat to Logstash

Let’s setup a simple Filebeat configuration to get the *.out files moving towards Elasticsearch. First stop: Logstash.

Change filebeat.yml to (comments etc. ommitted, I suggest you take the one from GitHub):

filebeat.prospectors:
- input_type: log
  paths:
    - <path where run.sh will run>*.out
  encoding: plain
  json.keys_under_root: true

output.logstash:
 # The Logstash hosts
 hosts: ["localhost:1337"]

logging.level: info

Configure in Logstash

A Logstash configuration consists of three parts – input, filter, output.

Input

Let’s connect to the Beats input on the host and port configured in the last part, and use json_lines as codec. This codec expects every line to be a new JSON object, exactly like our data is coded at the source.

Filter

We’ll add a couple of filter statements to alter the data before it goes into Elasticsearch:

  1. Because we’re running two types of tests, and iperf3 does not send different data other than 1 flag to indicate a download or upload test, let’s add a field indicating an upload or a download test.
  2. Some of the metrics come in bits, but Kibana supports nice human-readable formats for bytes, so we’ll divide the bits by 8 to express it in bytes.

Output

Let’s send the data to our Elasticsearch cluster using an index pattern with daily indices.

input {
  beats {
    port => 1337
    codec => "json_lines"
  }
}

filter {
  if [start][test_start][reverse] == 1 {
    mutate {
      add_field => {
        "test" => "download"
      }
    }
  } else if [start][test_start][reverse] == 0 {
    mutate {
      add_field => {
        "test" => "upload"
      }
    }
  }
  if [end][sum_received][bits_per_second] {
    ruby {
      code => "event['[end][sum_received][bytes_per_second]'] = event['[end][sum_received][bits_per_second]'] / 8"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => "localhost"
    index => "iperf3-%{+YYYY-MM-dd}"
    user => "<optional username>"
    password => "<optional password>"
  }
}

Set up Elasticsearch

Because we like to be lazy, ehr, efficient with our time, we’re not going to write out a new mapping from scratch. Instead, I’ve plainly sent some data to Elasticsearch and let it figure out a default mapping. The match was pretty good, but it can be better. Especially because we don’t care about full text searching in this data set, we’ll skip copying string fields to an analyzed and non-analyzed version next to each other, and opt for the non-analyzed one only. So I’ve taken the generated mapping, tweaked it a bit and used it to create an improved one and made it a template, too.

A template is just a mapping that is applied to any new index that matches it’s index name pattern. This will make sure that tomorrow’s index (remember that Elasticsearch loves splitting up time series data into daily indices!) will be created using our tweaked mapping template. It’s a bit long to paste the template here in full but it’s on the GitHub repo.

The easiest way to deploy the template is by going to Kibana Console (previously this was called Sense, and it was not part of the default installation of Kibana 4). You can take the contents of template.json and paste them directly in Kibana -> Console for execution.

Run it

Start Elasticsearch and Kibana according to their respective docs.

Start Logstash (assuming it’s in $PATH, otherwise point to […]/bin/logstash directly):

logtash -f logstash.conf -w1

Start Filebeat:

./filebeat -e -c filebeat.yml -d "publish"

Finally, start generating data:

./run.sh <N> <host> <port> <sleep>

Data should flow into the pipeline and end up in Elasticsearch. Kibana should pick up on it, too.

Use Kibana and Timelion to visualize the performance

Set up the Index Pattern and field formats

In order to have Kibana ‘see’ our data, we’ll need to point to it:

  1. Go to Management, Index Patterns, Add New, and use “iperf3-*” as the pattern
  2. Select “start.timestamp.time” as the time field. This will make sure the time of the benchmark is the time that counts as document time, not the time it was recorded in Elasticsearch.
  3. We need to set a field format. Find “end.sum_sent.bytes_per_second” in the list of fields and click the Edit icon. Select Bytes as the Format and then Update Field. This will make Kibana use human-readable formatting for this field and does not change the data itself.
Screen Shot 2016-07-31 at 11.37.28

Configure the index pattern in Kibana

Screen Shot 2016-07-31 at 11.38.10

Set two field to Bytes format

Set up some Kibana visualizations

Kibana has a feature that allows importing of dashboard and visualizations, which is what we’re going to use here. Get the file from GitHub and import it in Kibana 5 (Management -> Saved Objects -> Import). It should bring up a new dashboard with visuals.

Screen Shot 2016-07-31 at 20.44.34

Import dashboard and visualizations in Kibana

Ok awesome, we got some basic metrics and a download + upload histogram going. But, Timelion can do even more.

Go further with some Timelion charts

Timelion is perfect for this kind of time series metrics, because it allows us to plot multiple queries (from different indices) into a single chart, and even allows us to do arithmetic with a cool chainable query language. If you haven’t met Timelion yet, I recommend the short video and the tutorial.

Installing Timelion

Timelion, although open source & developed by Elastic, is not a standard module in Kibana. After stopping Kibana, we can install Timelion with a simple command:

bin/kibana-plugin install timelion

Then start Kibana normally. Timelion will allow us to use simple arithmetic in our visualizations, and also to combine multiple data sources (even from different clusters or external feeds like Quandl) right into Kibana.

Adding a Timelion Chart

In Kibana’s Timelion tab, you can add charts to a dashboard, similar to the way you add visualizations to a dashboard. Let’s create a new chart (it will start off by showing a histogram of all indices in Elasticsearch), and put in this query:

.es(index="iperf3-*", q="test:upload", metric=avg:end.sum_received.bits_per_second).movingaverage(position=left, window=10).divide(1048576).label("Moving average of upload speed in mbit/s").color("navy").lines(fill=1,width=2), .es(index="iperf3-*", q="test:upload", metric=avg:end.sum_received.bits_per_second).divide(1048576).label("Download speed in mbit/s").color("#757AFF").points(radius=5, fill=5), .es(index="iperf3-*", q="test:download", metric=avg:end.sum_received.bits_per_second).movingaverage(position=left, window=10).divide(1048576).label("Moving average of download speed in mbit/s").color("#FAC5E2").lines(fill=1,width=2), .es(index="iperf3-*", q="test:download", metric=avg:end.sum_received.bits_per_second).divide(1048576).label("Upload speed in mbit/s").color("#FF75BF").points(radius=5, fill=5)

Put this in as a on-liner and without the comments. It should bring up a chart with the four time series. It’s also part of the dashboard. Let’s take one of the four time series and take a closer look:

.es( #data source is Elasticsearch
 index="iperf3-*", #the index
 q="test:upload", #the query, get upload tests only
 metric=avg:end.sum_received.bits_per_second) #we need an average of this field
.movingaverage( #let's put this through a moving average algorithm
 position=left, #left-looking
 window=10) #looking back 10 data points
.divide(1048576) #divide, since the data is in bits and we like mbit
.label("Moving average of upload speed in mbit/s") #add a label
.color("navy") #and a color
.lines(fill=1,width=2), #and express is in a line with a bit of a fill (10%)
Screen Shot 2016-07-31 at 20.50.57

The Timelion chart

 

Screen Shot 2016-07-31 at 21.16.26

Timelion integrated with classic Kibana dashboard

Next steps

This post lazily uses an public, external iperf server. This probably leads to unreliable data and should you want to put this into production, it’s a good idea to look at setting up a private iperf server. You could put it in the cloud or in your on-prem data center, depending on what kind of benchmark you’re looking to run. Also I will happily look at Issues or PRs around my code on GitHub, perhaps to make it run on more platforms or to weed out some bugs.

An obvious application for this use case is measuring the network performance of various endpoints in the same network to find weak spots. Telecommunication companies are blamed for network performance issues that are often caused by bad WiFi as a result of badly placing the hardware in the building or using the wrong settings. Because Filebeat runs on Google Go, it will happily run on low-powered devices (such as routers) and smartphones (such as Android), creating insight in the difference between end-user device performance and router performance.

4 thoughts on “Monitor network performance with iperf3 and the Elastic Stack

  1. John Peterson

    Thank you for the great article! Ive been looking for a graphing solution for iperf3, and from the visualizations I see this looks great.
    I started following this to monitor between 1 centos 7 server and 3 centos7 hosts but I do have a few questions. This is my first endeavor into deploying an ELK stack so forgive me if these are simple questions I am overlooking.
    In the Send the data from disk with Filebeat to Logstash section, this is on the host machines correct? If so, is the run.sh script to be installed on the server as well as the host machines?
    For the logstash.yml file on github, would that be created in /etc/logstash/logstash.yml?
    When I tried to run logtash -f logstash.conf -w1 I received the following error, which may be due to an incorrect path definition on my part.
    /usr/share/logstash/bin$ bash logstash -f logstash.conf -w1
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
    Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
    15:04:35.658 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch – Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:[email protected]:9200/]}}
    15:04:35.677 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch – Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:[email protected]:9200/, :path=>”/”}
    log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    15:04:36.664 [[main]-pipeline-manager] WARN logstash.outputs.elasticsearch – Restored connection to ES instance {:url=>#}
    15:04:36.677 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch – Using mapping template from {:path=>nil}
    15:04:37.805 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch – Attempting to install template {:manage_template=>{“template”=>”logstash-*”, “version”=>50001, “settings”=>{“index.refresh_interval”=>”5s”}, “mappings”=>{“_default_”=>{“_all”=>{“enabled”=>true, “norms”=>false}, “dynamic_templates”=>[{“message_field”=>{“path_match”=>”message”, “match_mapping_type”=>”string”, “mapping”=>{“type”=>”text”, “norms”=>false}}}, {“string_fields”=>{“match”=>”*”, “match_mapping_type”=>”string”, “mapping”=>{“type”=>”text”, “norms”=>false, “fields”=>{“keyword”=>{“type”=>”keyword”}}}}}], “properties”=>{“@timestamp”=>{“type”=>”date”, “include_in_all”=>false}, “@version”=>{“type”=>”keyword”, “include_in_all”=>false}, “geoip”=>{“dynamic”=>true, “properties”=>{“ip”=>{“type”=>”ip”}, “location”=>{“type”=>”geo_point”}, “latitude”=>{“type”=>”half_float”}, “longitude”=>{“type”=>”half_float”}}}}}}}}
    15:04:37.859 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch – Installing elasticsearch template to _template/logstash
    15:04:38.593 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch – New Elasticsearch output {:class=>”LogStash::Outputs::ElasticSearch”, :hosts=>[#]}
    15:04:38.604 [[main]-pipeline-manager] INFO logstash.pipeline – Starting pipeline {“id”=>”main”, “pipeline.workers”=>1, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>5, “pipeline.max_inflight”=>125}
    15:04:41.520 [[main]-pipeline-manager] INFO logstash.inputs.beats – Beats inputs: Starting input listener {:address=>”0.0.0.0:1337”}
    15:04:41.744 [[main]-pipeline-manager] INFO logstash.pipeline – Pipeline main started
    15:04:42.076 [Api Webserver] INFO logstash.agent – Successfully started Logstash API endpoint {:port=>9601}
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
    Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
    15:06:10.210 [LogStash::Runner] FATAL logstash.runner – Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the “path.data” setting.
    And when I try to define an index pattern the time field never populates leading me to think the data is not being picked up on.

    I appreciate the work you put into this and I look forward to getting it set up!

    Best,

    Reply
    1. loek Post author

      It seems that you’ve tried to start Logstash twice, which is not allowed. Maybe try `ps aux | grep logstash` & `kill ` to kill off the hanging Logstash first before trying again?

      The other messages, including about logstash.yml, are probably okay to ignore. But you can try `–path.settings` as discussed at https://www.elastic.co/guide/en/logstash/5.4/running-logstash-command-line.html. This is a new Logstash message that didn’t exist when I wrote this article.

      Reply
    2. John Peterson

      I finally got this up and running, thank you for a great post!
      I did want to add that on the 26th line of the kibana.json file there is , missing after the }

      Also, would it be possible to add an entry to specify which host/beat the data was shipping from?

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.