The Beacon Experiments: Low-Energy Bluetooth Devices in Action

17 Feb 2014 The Beacon Experiments: Low-Energy Bluetooth Devices in Action

I was recently lucky enough to get a hold of a couple of sets of Bluetooth low-energy beacons; three Estimote beacons and a set of five Kontakt beacons. Whilst the Estimote beacons sure looked a lot fancier, I was keen to see if it was all just for show, or if they truly were a superior product. So I got to work putting them through their paces by conducting some controlled experiments to test their behaviour. Throughout the rest of this post I’m going to discuss the results of these experiments and conclude with some observations on their real-world usefulness for providing proximity-based services.

So What IS a Beacon?

Basically a beacon is a low energy bluetooth device that transmits packets of data that allow smart devices (phones, tablets, computers etc) to be informed when they are in range. Not only are the smart devices capable of being alerted when they are in range, but they are also capable of calculating their proximity to the beacons. The details of this second point are quite scarce and something I’ve been keen to explore further. I wanted to know just how precise this proximity was, and what factors could influence it’s accuracy.

What was I Working With?

As previously mentioned, I had two brands of beacons to play with, Estimote and Kontakt. Both can be purchased from their respective websites. I’m sure you could Google the products, but they’ll likely change over time, so here’s the photos of the exact beacons and packaging that I used in my experiments. I was also using an iPhone 5 running iOS 7.0.4.

Under the Hood

The first thing I wanted to do was see what was inside these beacons. So I grabbed a pen and pried one open.

Apart from a decent sized battery, and a small circuit board, there wasn’t much more to the Kontakt beacons. The Estimote beacons were a little harder to open as they seemed to be held together with a bucket load of glue. So I left them alone, though it did leave me wondering, how was I supposed to change the battery in two years when it went flat?

The Experiments

I wanted to determine whether I could use these beacons to determine a user’s location with relative precision, within about a 30cm radius of accuracy. I knew that the best way to do this would be to set up a grid of beacons and use some sort of triangulation algorithm to position the user inside the grid. But this all seemed a little too overwhelming to dive straight into. So I thought I’d strip it back, by devising some much smaller experiments and build it up from there. If I could get a better understanding of how the beacons behaved in isolation, I felt I’d have a better chance of designing a system of beacons working together to position a user within the system.

Using the iOS 7’s CoreLocation API is fairly simple, and there are plenty of tutorials on the web for how to get started. I began by writing an app that would simply display a table of ranged beacons, their RSSI (received signal strength indication), accuracy, and proximity values. Then as updates were received from the CLLocationManager, the app would refresh the details of the beacons in the table. This was fine for an entry point app, but anyone who’s used beacons before would know, the numbers fluctuate far too much for your eye to make any judgement of what is being represented. So I decided to modify my app slightly to save all the data and allow it to be exported as a csv, then it could be emailed directly from the iPhone. I could now run some controlled experiments, record all the data, and email it to myself after each experiment. This way I could plot the data and derive some conclusions about the beacons in different scenarios.

Experiment 1 – Testing the Beacons

My first experiment was to just record the beacon’s behaviour at a constant distance of 1.5 meters over the course of a minute. I wanted to look for differences between the two brands, as well as how consistent the beacons were within the same brand. In the following charts I’ve plotted the value of the ‘accuracy’ variable provided by the CoreLocation API. This is a value that I’ll explore in more detail later in this post, but for now all we’d expect to see from a stationary iPhone is a stable accuracy, that’s consistent amongst the same brand. I must say the results were a little surprising.

The first thing that can be seen is the differences between the two brands. Whilst the Kontakt beacons all hovered around the same accuracy level, the Estimote beacons were far more dispersed. The Estimote beacons varied from a maximum of 6.52 meters down to 1.48 meters, a differential of 5.04 meters. On the other hand, the Kontakt beacons differed in their accuracy by only 1.18 meters. I look further into why this was the case in Experiment 4 below.

Another observation of this experiment was the rate of dropouts between the two brands. The Estimote beacons did not dropout at all, however all the marks on the Kontakt chart along -1 meters are dropouts. This is where the API informs us that it knows it is in range of a beacon, however it has no idea of how far away, and hence no idea of the accuracy.

From this experiment I’d established that the Estimote beacons were somewhat inconsistent between themselves, though in isolation they did seem to be more reliable and have a steadier accuracy. Whilst the Kontakt beacons were a little more spasmodic they did seem to be more consistent amongst themselves.

Experiment 2 – Accuracy vs Distance

There is a fair bit of discussion on the web around the CLBeacon’s ‘accuracy’ property. There is an accepted Stack Overflow answer suggesting that this is just iOS speak for ‘distance’. In the Android port of the iBeacon library the accuracy variable is documented as “A double that is an estimate of how far the iBeacon is away in meters”. However the iPhone SDK documentation clearly states not to use it as a measure of distance from a beacon. This begs the question, why is it in meters? And why does the iPhone SDK documentation say “measured in meters from the beacon”?

Of course this property really has nothing to do with the beacons themselves, and is just each client’s own interpretation of the signal strength received. To test CoreLocation’s implementation of the accuracy property I set up an experiment where I moved a beacon backward by 50cm every minute whilst recording the accuracy. If accuracy is the same as distance, then I’d expect to see some sort of correlation with the actual distance the iPhone is away from the beacon.

At first glance it would appear that accuracy does seem to resemble distance. However at 0.5 meters, the accuracy underestimated distance, and by 1.5 meters, accuracy was an overestimation of distance. I decided to extend the distance out to 8 meters to see if the accuracy continued increasing with distance, however over the course of 80 seconds the accuracy of the beacon only reported an accuracy of 3.87m – some 4.13 meters shy of 8 meters. This was less than convincing that accuracy was representing distance.

In certain ranges with this beacon there does appear to be some correlation between accuracy and distance, but is this only coincidental? After all it does makes sense that accuracy would be improved the closer you are to a beacon, and would become worse the further away you are from a beacon. So although some correlation can be seen, can we rely with certainty on accuracy to be used as a measure of distance from a beacon?

Experiment 3 – Power Level

One of the cool features of these beacons is that you can configure some of their properties yourself. By using an iPhone app called BLExplr (available from the app store) you can get in and adjust the configuration of your Kontakt beacons. And by using the Estimote Virtual Beacon app (also in the app store) you can toy around with the Estimote beacons.

I began by doing what all good developers do… Cranking the power level (txPower) up to 100%! With the power level at it’s maximum I took my Kontakt beacons and ran a similar experiment to the previous one, where I would plot the accuracy at different distances. If the power level was higher, assumably the signal strength would be higher too and hence we’d expect a greater accuracy.

As suspected this appears to be the most convincing evidence, that accuracy has no relationship to distance. Notice that with the Kontakt beacon at 100% power (0dBm), the accuracy hovered just above 0 meters throughout the six minutes of the experiment. If accuracy was distance, you’d expect the calculation to take txPower into account and give us a value that much more closely resembled the actual distance.

To confirm this I decided to run the same experiment but with an Estimote beacon. I turned the power up to 100% (+4bBm) and measured the accuracy at the same varying distances.

Wow! This seemed to completely contradict the results seen from the Kontakt beacons. This time I was seeing a very close relationship between accuracy and distance between 1 and 5 meters, though it did start to slip at greater distances (however at 15 meters there were desks and computers obscuring the path, so perhaps this is expected). Why was the iPhone compensating for the increased txPower with the Estimote beacons, but with the Kontakt beacons it was not? Which was the correct interpretation? I decided to dig around further to determine whether accuracy could be trusted to represent distance like the Estimote beacons were showing, or if the Kontakt beacons were correct in showing improved accuracy because we had a higher power.

The answer lay with ‘measured power’. Measured power is the value of measured RSSI at a distance of one meter. It is passed by a beacon in it’s packets to help devices calculate distance based on their current RSSI. It is supposed to be altered by the beacons when txPower is set so that the receiving device knows how to interpret the power of the signal – whether it’s high due to increased power or high due to close proximity. The Estimote beacon knowledge base and the Kontakt beacon documentation claim to set this value in their becaon’s packets, though it appears my Kontakt beacons must have forgotten all about this. Now my iPhone thinks the high signal strength it’s receiving from the Kontakt beacons is as a result of close proximity not due to a high txPower.

So according to both the beacon brands, measured power is provided to help the client (in this case the iPhone) determine the distance of the device. And this is exactly what we are seeing with the Estimote beacons and the iPhone’s accuracy. However we are not seeing this with the Kontakt beacons, leading me to conclude that my Kontakt beacons are in fact faulty. After contacting the manufacturer of these beacons and describing the problem, they were unable to help, pointing out that the use of a third party app (BLExplr) was a potential cause of the problem. This was strange considering that this is the app they instructed me to use when I received the beacons. They did tell me that they will have their own app released soon, so perhaps when this is released, the problem will be solved.

 Experiment 4 – Beacon Inconsistency

As discussed measured power is controlled by your beacons, and cannot be altered by the user. This means that if configured incorrectly you can get vastly different accuracy values from one beacon to the next at the same distance. But are there other potential causes of inconsistency?

I decided to run an experiment where I would test the effects of rotation on my beacons. I would rotate the beacons by 90 degrees every 120 seconds, and if the orientation was playing a part in the beacon’s performance, then I’d expect to see a change in the beacon’s signal strength at different orientations.

It is clear from this chart that the orientation of the beacon was causing a change in the beacon’s signal strength. Notice that between 120 – 240 seconds (at 90 degrees rotation) this beacon’s signal strength was increased, and also much more consistent. To what degree this is changing the accuracy can be seen from the following chart, which is the exact same 480 seconds as the previous chart, but showing the calculated accuracy.

I’d also seen discussion on the web around the impact of closely grouped beacons. Is it possible that if beacons are positioned too close together that they could be interfering with one another? In order to test this, I decided to start recording a beacon’s signal strength whilst the beacon was in isolation. Then after every two minutes, I’d introduced a new beacon right next to the existing beacon. If interference was occurring I’d expect to see a noticeable change in the original beacon’s signal strength.

From this chart, we can see that there is no change in signal strength between the six minutes. In fact, the average signal strengths are -76.34, -75.92, -75.88. We do have a maximum of -72 db moments after the introduction of the final interference beacon, however this is not maintained, and -72 db is actually a stronger signal strength rather a degradation of signal strength. So it can be concluded that with three beacons, no interference is present.

Whilst interference is not playing a part in the beacon’s performance, rotation certainly is. It appears that the internal antenna must be favouring a particular direction, and this is having a major impact on the beacons. In a real world scenario it would be impractical to determine exactly which orientation each beacon should be facing for best performance. Further more, depending on how you are using your beacons you may need them to work through 360 degrees, so positioning of the antenna is irrelevant. This is just something to be aware of.

In the Real World

Having completed these experiments, I’ve been able to conclude that achieving accurate distance measurements with a single beacon is potentially possible, provided you have the right beacon and the right brand. Out of eight beacons, I only found one that was consistently achieving accurate distances. Given that there is so much inconsistency between the beacons and there is no guarantee you will even receive a beacon in an order that does provide accurate results, it seems the only way to achieve accuracy is by using many beacons and averaging the results.

So is it possible that by using multiple beacons each providing somewhat inaccurate data that we can deduce something more accurate? I’ve spent some time trying to achieve this, so please stay tuned for my upcoming blog post where I’ll discuss the results of these additional experiments, which involve tracking a moving iPhone through a grid of tactically placed beacons. I’ll also take a more in depth look at using beacons in the real world but now with a better understanding of how beacon signal strength, accuracy, and proximity should be interpreted.