Logs, metrics and what to log.

Lately I've had to decide (and give some advice)  on logging and metrics.  I'm learning a lot as I go, but I think I'm starting to get some kind of grip of what should be what, and on what level.

Here's this link about logs vs metrics, which makes lots of sense. 

And also, a post about logging levels, and how a very basic rule serves for it deciding which level means what. Very useful info in the associated reddit thread.

• DEBUG: information that is useful during development. Usually very chatty, and will not show in production.
• INFO: information you will need to debug production issues.
• WARN: someone in the team will have to investigate what happened, but it can wait until tomorrow.
• ERROR: Oh-oh, call the fireman! This needs to be investigated now! 

Also, tools: lnav , oklog.