How can I put data into DevicePilot efficiently?

Tips for optimising data ingestion

The following will help you make the most of DevicePilot's ingestion rate limits. None are absolute requirements.

  • Get DevicePilot to subscribe to your broker, rather than POSTing data in. Subscribing is particularly efficient for AWS-IoT Device Shadow because that only sends data to subscribers when any property changes, thus removing redundant repeats (we've seen this reduce data by 10:1)
  • Send multiple messages per POST, i.e. a JSON array like this: [{"$id":1, ...}, {"$id":2, ...}]. Include up to 500 messages per POST, limited by overall payload size.
  • Don't send more than about 1 message/device/minute, as there's very rarely any management value in higher-frequency data
  • To trial DevicePilot on a large estate of devices (10,000+ devices), reduce the number of devices to around 1,000 by picking a subset of devices (e.g. a random subset chosen by selecting on a hash of lower-order MAC address bits).