6th Jun
2010

Creating a Cacti Alternative

I want to replace cacti. I love graphs and information collation as much as the next guy, but I want something more and at the same time, something much less.

I have recently finished my second year of a Computer Science course at the University of Nottingham, and thus it is time to start thinking about a subject for my dissertation.

What is Cacti

Cacti "is a complete network graphing solution designed to harness the power of RRDTool's data storage and graphing functionality". It sports a web interface written in PHP and has a plugin type architecture.

Some examples of cacti graphs generated by RRDTool

I love what cacti can do, which is graphing the goings on of devices on your network. Cacti uses SNMP to retrieve this data, meaning anything SNMP can tell you about, cacti can track - and ultimately graph.

What's Wrong With Cacti?

Cacti's official website explains that "all of this is wrapped in an intuitive, easy to use interface". This is where issues begin to arise. Of course Cacti is a huge step up (interaction wise) from controlling RRDTool via the command line, however it is not what I would like it to be in terms of ease of use, and pure excitement factor.

Making graphs is one of the most exciting things you can do with a network of computers, and one of my favourite features is in fact the plugin Weathermap which has the potential to create fantastic diagrams of your network.

However I feel that cacti was developed in some sense for people who knew a lot about SNMP and RRDTool, and who just needed some help to manage their graphs. If you are like me, you see something which looks amazing and then learn how to do it, and Cacti does not make me welcome.

The Proposal

I want to construct a Cacti like contender from scratch using Django and it has to do several things differently:

  • Users do not have to understand SNMP or RRDTool
  • Graphs are not images, but generated from memcached (or similar) data using jQuery and a lovely json API (Django Piston)
  • We make user interface decisions to keep a simple first glance
  • Advanced users with SNMP knowledge should not be restricted by our assumptions
  • Usage mindset should be about the devices and not RRDTool

Using Django

Django will help - as is its strength - to speed along with the generic development. However there are some interesting points to consider:

Do I have to use RRDTool?

MySQL does not have the ability to store data in the way RRD files do, thus there are two options.

  1. Represent all time series data (which would be multiple files in the RRD format) as a single table in the database.
  2. Keep a reference to all RRD files in a table to provide an interface only.

People have tried to tackle this conflict before such as Graphite and their Whisper RRDTool replacement. And then there is also RRS which is more like the hybrid I am looking for.

My initial idea is to create a Django app which can represent an RRD file as an object which provides an interface to the file. There are a few python RRDTool libraries out their which could help achieve this, but I would be interested to know if anyone else had thought about this kind of integration - that of Django and RRDTool.

Marcus Whybrow is a student at the University of Nottingham's school of Computer Science. He is a musician, a photographer and a designer. He always has a project on the go and especially enjoys creating web applications using the brilliant Django framework, which he is putting into action at Aware Monitoring in his capacity as a summer intern in his dissertation project TinyGraph.

  • urban

    RRS is a a big performance problem when monitoring lots of parametrs from lots of hosts, this is the reason why we move out from collectd to zabbix (maybe zabbix has some good ideas about storing those data in database?).

  • http://marcuswhybrow.net Marcus Whybrow

    What exactly is zabbix? I have looked at their website and it seems to do everything you might need to do. But how does it store its information, do you know what technology they use?

  • http://twitter.com/boothead boothead

    I've been looking at this too, however my choice of tech stack is slightly different:

    Eventlet websockets and repoze.bfg for the real time web stuff, see http://github.com/boothead/rpz.websocket
    For the visualization stuff I've just found http://vis.stanford.edu/protovis/ which I'm totally blown away by.
    For the back end I was thinking about rabbitmq and mongodb, but will probably replace both with redis – the new publish/subscribe works really well and redis is blazingly fast!

    For a flexible webapp like this I think repoze definitely has the edge.

    Sounds like an interesting project though, do you have any links to source code yet?

  • http://marcuswhybrow.net Marcus Whybrow

    Wow, protovis looks amazing! RabbitMQ, MongoDB and Redis are all things that are in my arena of though right now and BFG looks like an interesting project I will definitely look into.

    I don't have any source code yet, but I hope to get started soon. Not sure what university policies are for allowing anyone in the world to see what would be my dissertation work, but I do like to make repositories on github.

  • http://twitter.com/boothead boothead

    Fair enough – I've made a start on a process monitoring app here http://github.com/boothead/multivisor which displays CPU and RAM usage for process running under supervisord, it's still using raphael for the graphs but I'll be porting it to protovis as soon as I can.

    The major advantage for repoze.bfg is that you don't just have a url and a callable: You also have a context object so you get a massive amount of flexibility in how you structure your app. In the example of multivisor you can actually have the same code that loads the page for a process also handle the websocket connection and real time updates to the client too. I love it! :-)

  • http://marcuswhybrow.net Marcus Whybrow

    Before starting any code I will have a full read of the documentation for repoze.bfg. I do love to learn new technologies, and their overview explains that they gain inspiration from Django and other frameworks, so I should feel at home.

  • http://david.ulevitch.com/ davidu

    The RRD performance issue of updating 1000's of RRD files has already been solved in a very elegant way that scales wonderfully.

    http://oss.oetiker.ch/rrdtool/doc/rrdcached.en….

    I still think Cacti is a beast, but not because of RRD.

    -David

  • richlv

    zabbix stores both configuration and monitoring data in a database – currently mysql, postgresql, oracle and sqlite are supported.

  • http://marcuswhybrow.net Marcus Whybrow

    Great idea, I have just read the and they sound like a great replacement to RRDTool.

    If I understand how MongoDB works then a collection is stored as a single ‘document’ meaning it is akin to the single RRD file. If this is the case the I think that is my favourite way to implement this.

  • m0n5t3r

    MongoDb capped collections may be a good replacement for RRDTool ;)

  • http://marcuswhybrow.net Marcus Whybrow

    Great idea, I have just read the capped collection docs and they sound like a great replacement to RRDTool.

    If I understand how MongoDB works then a collection is stored as a single 'document' meaning it is akin to the single RRD file. If this is the case then I think that is my favourite way to implement this.

  • http://www.kelvinism.com Kelvin Nicholson

    I've recently finished a project that stored various performance metrics (not CPU/MEM). When I chucked the data in a normal RDBMS I had to pay special attention to indexes, or a simple query of data over 30 days would spike the CPU. Go figure.

    boothead: I'm tired of always seeing your comments around the tubes suggesting such cool stuff for me to evaluate.

  • http://www.digitalprognosis.com Jeff Schroeder

    Google “mongodb ate my data”… It seems to me using another kv store like cassandra or even riak would be a much better idea.

  • http://security-wire.com/01/how-to-remove-windows-disk-fake-security-program.html remove windows disk

    Your article covers most of aspects we should be careful. Thank you!

  • Bruce Bushby

    ooooh eventlet and websockets …..now thats the business!

    Have you looked at Python Flask? I tried to get into Django and bfg…..they just seemed too big for what I wanted (small Intranet webapps for financial clients) …. step in Python Flask.

  • http://appletv.sundog.dk/ Frank

    Hey Marcus, how did it go, did you get started on the project? Sounds quite interesting with Protovis..

    /franksemi

  • http://marcuswhybrow.net Marcus Whybrow

    Hi Frank, I haven’t continued the project past its university life-time. I don’t have a personal need for it anymore either so I guess I have to declare the project asleep.

blog comments powered by Disqus