What is ELK (Elasticsearch, Logstash and Kibana)?

Can this tool suite be used to visualise and manage IoT?

ELK is an open-source toolchain, available to buy as-a-Service, designed to capture and analyse server logs. Logstash parses textual logs, elasticsearch is a database with excellent free-text retrieval, and Kibana is a charting tool.

Can one use ELK for IoT monitoring and management? A surprising number of people do attempt it, probably because they're already familiar with the stack from server monitoring. Certainly if you have a small number of devices (e.g. tens of devices) and just want to visualise their telemetry (like a software oscilloscope) then it's quite doable, though you may need to build a proprietary adaptor because ELK doesn't support IoT protocols natively.

However as your number of devices increases into the hundreds, and the questions that you need to answer go beyond looking at individual devices over short time periods to starting to measure performance over time, and metrics across groups of devices, the mismatch between ELK's design and IoT's requirements becomes apparent. The only way to measure e.g. uptime across your device estate on ELK is to write code which pre-processes the incoming telemetry to detect e.g. whether a device is up in real time, and write that conclusion into ELK, whereafter you can query it. The problem with that is:

a) You have to write code (and it's quite complex code, because for example if a device stops talking then after a timeout you'll want to record that the device has gone offline, even though there's no event to trigger running your code in that case)

b) You have to maintain the code, and change it every time anyone wants to ask different questions. Shouldn't you be spending your time on building your application?

c) You can't ask a new question on old data, because you're pre-processing at ingestion.

And remember that ELK isn't really a finished business application, it's a piece of technical infrastructure with a charting tool designed for use by engineers.

ELK can also be extremely expensive to use for IoT - we've seen examples of people with only circa 15k IoT devices spending in the region of $15,000/month on ELK. That's a problem because your average IoT device is worth much less than your average server, and there are generally a lot more IoT devices. Most people would consider an estate of 10,000 IoT devices to be quite small, but an estate of 10,000 servers to be quite large.

ELK is a great solution, but to a different problem.