As mentioned before we run a four-node cluster for redundancy and charding. A pair of two nodes persists all metrics. To see if there is anything wrong, there is a monitoring check testing for double metrics, where they shouldn’t be. One day we got an alert reporting 20 double metrics. After some research we found out, that they all come from one application, where there was a self-implemented graphite-connector creating metrics for requests depending on the request uri. An error inside this caused creation of metrics named like „a..b“ and „a.b“. The double-dottet metrics pass carbon-relay and the consistent-hash algorithm sends them to a different cache than the single-dottet ones. Carbon-cache removes those double-dots and converts „a..b“ to „a.b“. This leads to a whisper file created for „a.b“ on both hosts. Voila, a problem was born.

IMG_1338

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

*

Diese Website verwendet Akismet, um Spam zu reduzieren. Erfahre mehr darüber, wie deine Kommentardaten verarbeitet werden .