Ça c'est une très bonne nouvelle !

!/bin/bash

How to install Graphite as a user on CentOS 5.5 (with neither internet access and root account)

--------------------------------------------------------------------------------

Tomasz Kalkosiński - refaktor.blogspot.com

#

Graphite is a Scalable Realtime Graphing (http://graphite.wikidot.com/)

This script is based on two excellent tutorials:

http://community.webfaction.com/questions/10038/how-to-install-pycairo-in-python27-thanks

https://gist.github.com/jgeurts/3112065

My scenario:

- no internet access

- only user account, no root account

- gcc and make installed

- some libraries missing

- no development packages for installed libraries

#

Goals:

- installation directory is $HOME/graphite-install

- install libraries to $HOME

- install Graphite and stuff to $HOME/graphite

#

If you see wget here - wget it to your local machine and upload to remote CentOS.

You must have gcc and make installed. You can't go on without it.

Check it with:

rpm -qa | grep gcc

---- 1. Environment setup ----

Set up these variables. Some of these directories don't exist yet. They will be created as you go on.

You can put these variables to your .bashrc.

export CFLAGS="-I$HOME/include -I$HOME/include/python2.7"
export CPPFLAGS="-I$HOME/include -I$HOME/include/python2.7"
export PKG_CONFIG_PATH="$HOME/lib/pkgconfig"
export LDFLAGS="-L$HOME/lib -L$HOME/lib/python2.7"
export LD_LIBRARY_PATH="$HOME/lib:$HOME/lib/python2.7:$HOME/lib/python2.7/site-packages/cairo"
export PYTHONPATH="$HOME/lib/python:$HOME/lib/python2.7/site-packages:$HOME/.local/bin/usr/local/lib/python2.7/site-packages"
export PATH="$HOME/.local/bin:$PATH"

---- 2. Libraries ----

These libraries are required by Graphite. You need to compile them, because other libraries and python modules

need their headers. They are installed to $HOME/lib and headers are placed in $HOME/include.

cd $HOME/graphite-install

wget http://zlib.net/zlib-1.2.8.tar.gz
wget ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-1.6.2.tar.gz
wget http://www.sqlite.org/2013/sqlite-autoconf-3071700.tar.gz

tar zxf zlib-1.2.8.tar.gz
tar zxf libpng-1.6.2.tar.gz
tar zxf sqlite-autoconf-3071700.tar.gz

cd zlib-1.2.8
./configure --prefix=$HOME
make
make install

cd ../libpng-1.6.2
./configure --prefix=$HOME
make
make install

cd ../sqlite-autoconf-3071700.tar.gz
./configure --prefix=$HOME
make
make install

---- 3. Python 2.7.5 ----

Python installed by default in CentOS may miss some native modules compiled. You need to recompile.

cd $HOME/graphite-install
wget http://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz

tar zxf Python-2.7.5.tgz
cd Python-2.7.5

enable-shared is crucial here

./configure --enable-shared --prefix=$HOME
make

Check output for this line:

Python build finished, but the necessary bits to build these modules were not found:

Make sure that _zlib and _sqlite3 IS NOT on that list. Otherwise you miss dependencies or you had errors before.

make install

Now you can have two Python installations in system. I use ~/bin/python everywhere from now on to use my version.

---- 4. Cairo ----

Cairo has some dependencies to fulfill.

cd $HOME/graphite-install
wget http://cairographics.org/releases/pixman-0.26.2.tar.gz
wget ftp://sourceware.org/pub/libffi/libffi-3.0.11.tar.gz
wget http://ftp.gnome.org/pub/GNOME/sources/glib/2.31/glib-2.31.22.tar.xz
wget http://cairographics.org/releases/cairo-1.12.2.tar.xz
wget http://cairographics.org/releases/py2cairo-1.10.0.tar.bz2

tar xzf pixman-0.26.2.tar.gz
tar xzf libffi-3.0.11.tar.gz

My system didn't have unxz so I had to download and repack to zip.

unzip glib-2.31.22.zip

unzip cairo-1.12.2.zip

unxz glib-2.31.22.tar.xz
unxz cairo-1.12.2.tar.xz
tar xjf py2cairo-1.10.0.tar.bz2

cd libffi-3.0.11
./configure --prefix=$HOME
make
make install

cd ../glib-2.31.22
./configure --prefix=$HOME
make
make install

cd ../pixman-0.26.2
./configure --prefix=$HOME
make
make install

cd ../cairo-1.12.2
./configure --prefix=$HOME
make
make install

cd ../py2cairo-1.10.0
~/bin/python ./waf configure --prefix=$HOME
~/bin/python ./waf build
~/bin/python ./waf install

Check if cairo is properly installed:

~/bin/python -c 'import cairo; print cairo.version'

You should see cairo version number.

If there is an import error, there is something wrong with your PYTHONPATH.

---- 5. Django and modules ----

Remember that it is all installed in user directory.

cd $HOME/graphite-install
wget https://django-tagging.googlecode.com/files/django-tagging-0.3.1.tar.gz
wget https://www.djangoproject.com/m/releases/1.5/Django-1.5.1.tar.gz
wget https://pypi.python.org/packages/source/z/zope.interface/zope.interface-4.0.5.zip#md5=caf26025ae1b02da124a58340e423dfe
wget http://twistedmatrix.com/Releases/Twisted/11.1/Twisted-11.1.0.tar.bz2

unzip zope.interface-4.0.5.zip
tar zxf django-tagging-0.3.1.tar.gz
tar zxf Django-1.5.1.tar.gz
tar jxf Twisted-11.1.0.tar.bz2

cd zope.interface-4.0.5
~/bin/python setup.py install --user

cd ../Django-1.5.1
~/bin/python setup.py install --user

cd ../django-tagging-0.3.1
~/bin/python setup.py install --user

cd ../Twisted-11.1.0
~/bin/python setup.py install --user

---- 6. Graphite ----

Remember that it is all installed into $HOME/graphite directory!

cd $HOME/graphite-install
wget https://launchpad.net/graphite/0.9/0.9.10/+download/graphite-web-0.9.10.tar.gz
wget https://launchpad.net/graphite/0.9/0.9.10/+download/carbon-0.9.10.tar.gz
wget https://launchpad.net/graphite/0.9/0.9.10/+download/whisper-0.9.10.tar.gz
tar -zxvf graphite-web-0.9.10.tar.gz
tar -zxvf carbon-0.9.10.tar.gz
tar -zxvf whisper-0.9.10.tar.gz

cd whisper-0.9.10
~/bin/python setup.py install --home=$HOME/graphite

cd ../carbon-0.9.10
~/bin/python setup.py install --home=$HOME/graphite

cd ../graphite-web-0.9.10

You're almost there. Check if all dependencies are met:

~/bin/python check-dependencies.py

There should be no fatal errors, only warnings for optional modules.

~/bin/python setup.py install --home=$HOME/graphite

---- 7. Configuration ----

cd $HOME/graphite/conf

Copy example configurations

cp carbon.conf.example carbon.conf
cp storage-schemas.conf.example storage-schemas.conf
vim storage-schemas.conf

You should really read document on how to configure your storage. It is a quick read:

http://graphite.wikidot.com/getting-your-data-into-graphite

Configure it the way you need.

cd $HOME/graphite/lib/python/graphite
cp local_settings.py.example local_settings.py

vim local_settings.py

This file is crucial:

Line around 13: setup your TIME_ZONE from this list: http://stackoverflow.com/questions/13866926/python-pytz-list-of-timezones

Line around 23: DEBUG = True

Line around 24: append ALLOWED_HOSTS = ["*"]

Lines around 46-54: change paths from /opt/graphite to your explicit graphite installation dir,

for example /home/you/graphite. $HOME doesn't work here!

Uncomment lines 141-150 (starting with DATABASES) and update NAME with full explicit database path. $HOME doesn't work here!

Create new database for Graphite web application

~/bin/python manage.py syncdb

Start carbon deamon

cd $HOME/graphite
~/bin/python carbon-cache.py status
~/bin/python carbon-cache.py start

I run development graphite server in a screen, since I cannot expose it via httpd.

screen

In screen

~/bin/python manage.py runserver

Detach with Control-A, Control-D (attach again with screen -r next time)

Check if it works, you should see <title>Graphite Browser</title>

curl localhost:8000

You can run some example client to send some stats.

You can adjust example-client.py delay value for more/less stats.

cd $HOME/graphite/examples
~/bin/python example-client.py

Done!

Easy as pie :)

curl -X POST "http://localhost/events/" -d '{"what": "Web Service", "tags": "production deploy", "data":"version 1.1.7"}'

La base de la métrologie avec les différents types de metric : gauge, rate, histogram.. Le rôle de statsd (buffer..)

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister

used by Doo 1M metric/sec (12 CPU)

Scaling graphite
via arnaudb

Un get la dessus : /api/datasources/proxy/1/metrics/find/?query=collectd.*

Il existe plusieurs alternatives à carbon, la plus connu étant surement influxdb. Celle ci est basée sur cassandra!

+https://github.com/kairosdb/kairos-carbon

via arnaudb

Un screenshot d'un dashboard qu'il est bien

! /bin/sh

BEGIN INIT INFO

Provides: carbon-cache

Required-Start: $remote_fs $syslog

Required-Stop: $remote_fs $syslog

Default-Start: 2 3 4 5

Default-Stop: 0 1 6

Short-Description: carbon-cache init script

Description: An init script for Graphite's carbon-cache daemon.

END INIT INFO

Author: Jeremy Chalmer

#

This init script was written for Ubuntu 11.10 using start-stop-daemon.

Note: Make sure you set the USER field in /opt/graphite/conf/carbon.conf to be the same

user that owns the /opt/graphite/storage/ folder. Carbon-cache will be invoked as that

username on start.

#

Enable with update-rc.d carbon-cache defaults

Source init-functions:

#source /lib/lsb/init-functions
. /lib/lsb/init-functions

PATH should only include /usr/* if it runs after the mountnfs.sh script

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Path to Graphite

GRAPHITE_HOME=/opt/graphite

Name of executable daemon

NAME=carbon-cache
DESC=carbon-cache

#Carbon has its own logging facility, by default in /opt/graphite/storage/log/carbon-cache-*

Path to Executable

DAEMON=$GRAPHITE_HOME/bin/carbon-cache.py

NOTE: This is a hard-coded PID file, based on carbon-cache.py. If you have more the one carbon-cache

instance running on this machine, you'll need to figure out a better way to calculate the PID file.

PIDFILE=/opt/graphite/storage/carbon-cache-a.pid

SCRIPTNAME=/etc/init.d/$NAME

Exit if the package is not installed

if [ ! -x "$DAEMON" ]; then {
echo "Couldn't find $DAEMON or not executable"
exit 99
}
fi

Load the VERBOSE setting and other rcS variables

[ -f /etc/default/rcS ] && . /etc/default/rcS

#

Function that starts the daemon/service

#
do_start()
{

Return

#   0 if daemon has been started
#   1 if daemon was already running
#   2 if daemon could not be started

    # Test to see if the daemon is already running - return 1 if it is. 
start-stop-daemon --start --pidfile $PIDFILE \
    --exec $DAEMON --test -- start > /dev/null || return 1

    # Start the daemon for real, return 2 if failed
start-stop-daemon --start --pidfile $PIDFILE \
    --exec $DAEMON -- start > /dev/null || return 2

}

#

Function that stops the daemon/service

#
do_stop() {

Return

#   0 if daemon has been stopped
#   1 if daemon was already stopped
#   2 if daemon could not be stopped
#   other if a failure occurred
log_daemon_msg "Stopping $DESC" "$NAME"
start-stop-daemon --stop --signal 2 --retry 5 --quiet --pidfile $PIDFILE
RETVAL="$?"
[ "$RETVAL" = 2 ] && return 2

    # Delete the exisitng PID file
if [ -e "$PIDFILE" ]; then {
    rm $PIDFILE
}
    fi

    return "$RETVAL"

}

Display / Parse Init Options

case "$1" in
start)
[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
do_start
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
restart)
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
) log_end_msg 1 ;; # Failed to start
esac
;;
)

Failed to stop

log_end_msg 1
;;

les slides :
http://fr.slideshare.net/kennydee/monitoring-applicatif-pourquoi-et-comment

Aide à comprendre pas mal de chose concernant graphite/statsd

!/usr/bin/env python

-- coding: utf-8 --

def archive_to_bytes(archive):
def to_seconds(s):
SECONDS_IN_A = {
's': 1,
'm': 1 60,
'h': 1 60 60,
'd': 1 60 60 24,
'y': 1 60 60 24 365,
}

    return int(s[:-1]) * SECONDS_IN_A[s[-1]]

archive = [map(to_seconds, point.split(':'))
           for point in args.archive.split(',')]

SIZE_METADATA = 2 * 4 + 4 + 4  # 16 [!2LfL]
SIZE_ARCHIVE_INFO = 3 * 4  # 12 [!3L]+
SIZE_POINT = 4 + 8  # 12 [!Ld]+

size = 0
for resolution, retention in archive:
    size += SIZE_ARCHIVE_INFO + SIZE_POINT * retention/resolution

if size:
    size += SIZE_METADATA

return size

if name == 'main':
import argparse

parser = argparse.ArgumentParser(
    description="Calculates the size of the whisper storage for the given \
            archive (in resolution:retention format, e.g. 1m:24h,5m:3m)"
)
parser.add_argument(
    'archive',
    help="Archive in storage-schemas.conf format (resolution:retention)"
)

args = parser.parse_args()

print "{} >> {} bytes".format(args.archive, archive_to_bytes(args.archive))

Et encore 1!

via arnaudb

In order to save graphs under 'My Graphs' and 'User Graphs' you simply need to log into Graphite and the Save/Delete buttons will then appear on the top of the Composer window. To log in you need to either create an account for yourself in the Django database or setup LDAP authentication. LDAP authentication is setup through local_settings.py, check out the example file's comments for details on doing that (or just ask if you run into any issues). However if you're not using LDAP its pretty easy to create a local user in Django's database using the Django admin interface.

When you first installed Graphite there was a step where you had to initialize the database by running 'manage.py syncdb'. That prompted you to create an admin user, if you did that already you're good to go. Otherwise you need to create an admin user by running 'manage.py createsuperuser'.

Once your admin user is setup you can go to the django admin interface by visiting http://my-graphite-server/admin/ (note: the trailing slash is required!) and logging in as your admin user. From there you need to create a new user account by clicking 'Add' under the 'Users' section. After that, go back to the main Graphite page. You may need to logout if it thinks you're still the admin user. Then log back in as you're new user and you should be able to save graphs! Also once logged in you have the ability to set profile options. Currently there is only 1 option, the "advanced UI features" option. Enabling that puts a '*' element in every folder that has more than one element, making it easier to use wildcards when you build graphs.

Hope that helps.

dash auto via collectd dashgen

via arnaudb

http://grey-boundary.com/the-architecture-of-clustering-graphite/
http://adminberlin.de/graphite-scale-out/
http://graphite.readthedocs.org/en/latest/carbon-daemons.html
http://blog.xebia.fr/2013/05/29/graphite-les-bases/
http://fr.slideshare.net/AnatolijDobrosynets/graphite-cluster-setup-blueprint

via arnaudb

Avec ça possible de faire des templates de graph

via skunnyk

why not

via arnaudb

Un article du tech blog de flickr sur la metrologie (qui a servi d'inspiration à l'écriture de statsd)

Le combo statsd/graphite a l'air vraiment puissant

Une présentation chez M6Tech concernant le monitoring (graphite inside)

encore un frontend graphite
via arnaub

Un frontend pour graphite, sympa!
via Skunnyk

Comment Feed carbon (le backend de graphite)

• https://graphite.readthedocs.org/en/latest/tools.html

statsd + graphite
Mais statsd en nodejs..