Pour remplacer graphite-web (django) par plus léger
Ça c'est une très bonne nouvelle !
TIL :
- wsp files has a length limit.
- use gauge in type, dont fucking use counter (it accepts only integer)
[Summary, full note in Shaarli]
Install Graphite 0.9.10 as a normal user on CentOS 5.5, with no internet and no root (Tomasz Kalkosiński, refaktor.blogspot.com). Based on https://community.webfaction.com/questions/10038 and https://gist.github.com/jgeurts/3112065. Download every tarball elsewhere and upload it. Needs gcc and make (rpm -qa | grep gcc).
curl -X POST "http://localhost/events/" -d '{"what": "Web Service", "tags": "production deploy", "data":"version 1.1.7"}'
On envoie une valeur aléatoire toutes les 10 secondes et on observe le résultat sur un dashboard grafana qui se refresh tout seul toutes les 10 secondes ;-)
for i in {1..60}; do line=$(echo "toto $(( ( RANDOM % 10 ) + 1 )) $(date +%s)"); echo $line; echo $line | nc -q0 localhost 2003; sleep 10; done
La base de la métrologie avec les différents types de metric : gauge, rate, histogram.. Le rôle de statsd (buffer..)
cd /opt/graphite/storage/whisper
find . -name "*.wsp" | xargs -I {} sh -c "echo -n '{} ' ; whisper-info.py {}|grep aggre|cut -d' ' -f2" > aggreg.dump
82024 average
2784 max
1378 min
8732 sum
59682 average
Donc oui, tous les whispers collectd sont en average.
Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister
used by Doo 1M metric/sec (12 CPU)
Scaling graphite
via arnaudb
[07:27] < torkelo>| matejz: I have managed to get about 140~ bytes per measurement (ES) asyd
[07:27] < matejz>| and was thinking of using it for metrics as well aviau
[07:27] < torkelo>| which is 12x the size requirement of Graphite (12 bytes per measurement)
[10:16] < torkelo> | agree, if you store more than 100 000 metrics/s I think ES is not a good option. But for short term performance logging the new metric features for flat_white
percentile and moving average, etc are looking very good
Un get la dessus : /api/datasources/proxy/1/metrics/find/?query=collectd.*
Il existe plusieurs alternatives à carbon, la plus connu étant surement influxdb. Celle ci est basée sur cassandra!
+https://github.com/kairosdb/kairos-carbon
via arnaudb
Encore une discussion intéressante sur les perf de graphite. Ce qu'il faut retenir c'est que le botteneck peut se situer au niveau du CPU ou au niveau du disque (la RAM en général ce n'est pas un probleme, meme si bien sur, il faut la surveiller)
Pour connaitre l'utilisation du CPU de carbon-cache, une metric est envoyé par le daemon dans carbon.agents.graphite-x.cpuUsage
Pour connaitre l'utilisation du disk, on se sert de iostat -dmx 1 2 (merci arnaud)
Si le disque est trop haut (entre 50 et 75), il faut le soulager en baissant dans la conf de carbon le max update par seconde.
Ce qui aura pour effet d'augmenter la taille du cache et donc de faire plus travailler le CPU..
Au contraire si le CPU est chargé mais que le disque ne fait rien, il faut augmenter le max update par seconde.
En trouvant le bon équilibre on peut exploiter au maximum le hardware disponible
Un screenshot d'un dashboard qu'il est bien
à lire pour le monitoring applicatif
front end graphite avec un backend cassandra (carrément)
via arnaudb
[Summary, full note in Shaarli]
Init script (/etc/init.d/carbon-cache) for Graphite's carbon-cache daemon, by Jeremy Chalmer, written for Ubuntu 11.10 with start-stop-daemon.
à regarder à la rentrée
via arnaudb
apply function > Special > draw non zero as infinite
Aide à comprendre pas mal de chose concernant graphite/statsd
Suite d'outils bien pratique pour tester/vérifier/debug ses whisper files, en particulier whisper-dump et whisper-fetch
def archive_to_bytes(archive):
def to_seconds(s):
SECONDS_IN_A = {
's': 1,
'm': 1 * 60,
'h': 1 * 60 * 60,
'd': 1 * 60 * 60 * 24,
'y': 1 * 60 * 60 * 24 * 365,
}
return int(s[:-1]) * SECONDS_IN_A[s[-1]]
archive = [map(to_seconds, point.split(':'))
for point in args.archive.split(',')]
SIZE_METADATA = 2 * 4 + 4 + 4 # 16 [!2LfL]
SIZE_ARCHIVE_INFO = 3 * 4 # 12 [!3L]+
SIZE_POINT = 4 + 8 # 12 [!Ld]+
size = 0
for resolution, retention in archive:
size += SIZE_ARCHIVE_INFO + SIZE_POINT * retention/resolution
if size:
size += SIZE_METADATA
return size
if name == 'main':
import argparse
parser = argparse.ArgumentParser(
description="Calculates the size of the whisper storage for the given \
archive (in resolution:retention format, e.g. 1m:24h,5m:3m)"
)
parser.add_argument(
'archive',
help="Archive in storage-schemas.conf format (resolution:retention)"
)
args = parser.parse_args()
print "{} >> {} bytes".format(args.archive, archive_to_bytes(args.archive))
[Summary, full note in Shaarli]
whisper-dump.py : script Python 2 qui affiche le contenu brut d'un fichier whisper (.wsp) de Graphite. Usage : whisper-dump.py path
In order to save graphs under 'My Graphs' and 'User Graphs' you simply need to log into Graphite and the Save/Delete buttons will then appear on the top of the Composer window. To log in you need to either create an account for yourself in the Django database or setup LDAP authentication. LDAP authentication is setup through local_settings.py, check out the example file's comments for details on doing that (or just ask if you run into any issues). However if you're not using LDAP its pretty easy to create a local user in Django's database using the Django admin interface.
When you first installed Graphite there was a step where you had to initialize the database by running 'manage.py syncdb'. That prompted you to create an admin user, if you did that already you're good to go. Otherwise you need to create an admin user by running 'manage.py createsuperuser'.
Once your admin user is setup you can go to the django admin interface by visiting http://my-graphite-server/admin/ (note: the trailing slash is required!) and logging in as your admin user. From there you need to create a new user account by clicking 'Add' under the 'Users' section. After that, go back to the main Graphite page. You may need to logout if it thinks you're still the admin user. Then log back in as you're new user and you should be able to save graphs! Also once logged in you have the ability to set profile options. Currently there is only 1 option, the "advanced UI features" option. Enabling that puts a '*' element in every folder that has more than one element, making it easier to use wildcards when you build graphs.
Hope that helps.
dash auto via collectd dashgen
via arnaudb
Je me mets ça de côté : la retention par défaut des RRD dans munin au format carbon :
[munin_schema]
pattern = ^munin.
retentions = 5m:2d,30m:10d,2h:45d,1d:1y
Sur les 2 derniers jours on a une mesure toute les 5 minutes
Sur les 10 derniers jours, une mesure toute les 30 minute est conservée
etc
/opt/graphite/conf/storage-schemas.conf
http://grey-boundary.com/the-architecture-of-clustering-graphite/
http://adminberlin.de/graphite-scale-out/
http://graphite.readthedocs.org/en/latest/carbon-daemons.html
http://blog.xebia.fr/2013/05/29/graphite-les-bases/
http://fr.slideshare.net/AnatolijDobrosynets/graphite-cluster-setup-blueprint
via arnaudb
If you have lots of metric names that change (new servers etc) in a defined pattern it is irritating to constantly have to create new dashboards.
With scripted dashboards you can dynamically create your dashboards using javascript. In the folder grafana install folder app/dashboards/ there is a file named scripted.js. This file contains an example of a scripted dashboard. You can access it by using the url:
http://grafana_url/#/dashboard/script/scripted.js?rows=3&name=myName
If you open scripted.js you can see how it reads url paramters from ARGS variable and then adds rows and panels.
Avec ça possible de faire des templates de graph
Pourquoi utiliser statsd en plus de graphite ?
Pour scale ? car stats reçoit en udp, attend un moment, puis envoie à graphite.
Mais Carbon semble pouvoir recevoir lui aussi directement en udp..
Pour répartir la charge tout simplement ? à voir..