Ganglia Graphite

Graphite - Ganglia integration

Graphite is an enterprise-scale monitoring tool. It has a powerful graph composer and lots functions to transform data. There are two major ways of integrating Ganglia with Graphite

• Ganglia gmetad sends metrics directly to Graphite's Carbon engine
• Graphite supports reading RRD databases directly

Shipping metrics to Carbon Server

As of Ganglia 3.3.0 gmetad supports sending Ganglia metrics directly to Graphite to be processed by Carbon and stored in Graphite's storage engine of choice e.g. whisper, cyanite etc.

To configure graphite in /etc/ganglia/gmetad.conf you will need to set following values

Server address or IP of your Carbon server

carbon_server "my.graphite.box"

If you are running carbon on a non-standard port you can adjust it with

carbon_port 2003

Carbon Protocol defaults to TCP. For performance reasons you may want to consider switching to UDP.

carbon_protocol udp

If you do end up using TCP default timeout to establish connection to graphite is 500 ms.

carbon_timeout 500

Graphite Prefix is what metrics are being prefixed with before being sent to Graphite. For example for load_one on host webserver1 and cluster webcluster will become

unspecified.webcluster.webserver1.load_one

To override that specify graphite prefix e.g.

graphite_prefix "datacenter1.gmetad"

Which will result in above metric being displayed as

datacenter1.gmetad.webcluster.webserver1.load_one

Point Graphite to Ganglia RRDs

If Ganglia Web is your primary interface it may quite advantageous to forgo installation of Carbon and use Ganglia RRDs directly.

Following directions assume you are using Ubuntu 14.04 since it comes with graphite-web already packaged.

Install packages

First we need to install graphite-web and python-rrd libraries to allow Graphite to read RRDs

apt-get install python-rrdtool graphite-web

Once the install is completed you will need to edit /etc/graphite/local_settings.py and add following content

SECRET_KEY = 'SECRET_KEY_SET_ME_PLEASE'
LOG_RENDERING_PERFORMANCE = True
LOG_CACHE_PERFORMANCE = True
LOG_METRIC_ACCESS = True
FLUSHRRDCACHED = 'unix:/tmp/rrdcached.limited.sock'
GRAPHITE_ROOT = '/usr/share/graphite-web'
CONF_DIR = '/etc/graphite'
STORAGE_DIR = '/var/lib/graphite/whisper'
CONTENT_DIR = '/usr/share/graphite-web/static'
RRD_DIR = '/var/lib/ganglia/rrds'
DATA_DIRS = [RRD_DIR]
LOG_DIR = '/var/log/graphite'
INDEX_FILE = '/var/lib/graphite/search_index'  # Search index file
DATABASES = {
    'default': {
        'NAME': '/var/lib/graphite/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

Things you may need to change are FLUSHRRDCACHED which is the socket used by your RRDcached instance. Another thing that may be different is location of RRDs. By default Ganglia puts them in /var/lib/ganglia/rrds. Change RRD_DIR to point to the right location.

Once you have configured local settings you will need to initialize Graphite Web's Django DB e.g.

graphite-manage syncdb
chown -R _graphite /var/lib/graphite

Follow the directions. This will initialize the Sqlite3 DB in /var/lib/graphite/graphite.db.

Last but not least we need to configure you web server. I am using mod_wsgi with Apache. This is a quick way to do it

apt-get install apache2 libapache2-mod-wsgi
ln -s /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled/headers.load
ln -s /etc/apache2/mods-available/wsgi.* /etc/apache2/mods-enabled

You will then need to copy and paste following snippet into /etc/apache/sites-enabled/002-graphite.conf

Listen 8013
<VirtualHost *:8013>

        # Set these headers so we can use it with e.g. Grafana
	Header set Access-Control-Allow-Origin "*"
	Header set Access-Control-Allow-Methods "GET, OPTIONS"
	Header set Access-Control-Allow-Headers "origin, authorization, accept"
	WSGIDaemonProcess _graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120 user=_graphite group=_graphite
	WSGIProcessGroup _graphite
	WSGIImportScript /usr/share/graphite-web/graphite.wsgi process-group=_graphite application-group=%{GLOBAL}
	WSGIScriptAlias / /usr/share/graphite-web/graphite.wsgi

	Alias /content/ /usr/share/graphite-web/static/
	<Location "/content/">
		SetHandler None
	</Location>

	ErrorLog ${APACHE_LOG_DIR}/graphite-web_error.log

	# Possible values include: debug, info, notice, warn, error, crit,
	# alert, emerg.
	LogLevel warn

	CustomLog ${APACHE_LOG_DIR}/graphite-web_access.log combined

</VirtualHost>

You should now be able to access Graphite web over http://server_name:8013/. If you get any Internal Server errors please look in the Graphite Web error log

/var/log/apache2/graphite-web_error.log

You should be pretty much done at this point except for one small detail. In general Ganglia uses fully qualified domain names for most storage so for example you will have this kind of directory structure in /var/lib/ganglia/rrds

/var/lib/ganglia/rrds/WEB/web1.domain.com
/var/lib/ganglia/rrds/WEB/web2.domain.com
/var/lib/ganglia/rrds/DB/db1.domain.com

Problem lies in the fact that Graphite uses dots as namespace separator so if you look at the hierarcy you will see web1.domain.com show up as

web1
    ->domain
            ->com

which will not work correctly. To get around that I have written a small script that you may want to run out of cron every e.g. 10 minutes that creates another directory structure e.g. /var/lib/graphite/rrds which is linked to /var/lib/ganglia/rrds however replaces dots with underscores so you end up with

/var/lib/graphite/rrds/WEB/web1_domain_com
/var/lib/graphite/rrds/WEB/web2_domain_com
/var/lib/graphite/rrds/DB/db1_domain_com

Script is here

#!/bin/bash

GANGLIA_RRDS="/var/lib/ganglia/rrds"
GRAPHITE_RRDS="/var/lib/graphite/rrds"

for cluster_dir in `ls -1d $GANGLIA_RRDS/*`
do

   CLUSTER=${cluster_dir##*/}
   GRAPHITE_DIR_NAME="$GRAPHITE_RRDS/$CLUSTER"
   # Check that cluster dir exists on Graphite site. If not make it
   if [ ! -d $GRAPHITE_DIR_NAME ]; then
     mkdir -p $GRAPHITE_DIR_NAME
   fi
   for file in `ls -1d $cluster_dir/*` 
   do

     host_name=${file##*/}
     host_modified=`echo $host_name |tr . _`
     LINK_DEST="$GRAPHITE_RRDS/$CLUSTER/$host_modified"
 
     if [ ! -s $LINK_DEST ]; then 
        ln -s $file $LINK_DEST
     fi

   done

done

Now you just have to change /etc/graphite/local_settings.py to point to /var/lib/graphite/rrds. Set RRD_DIR to that.

That's all folks. Enjoy it.