The Beacon Experiments: Low-Energy Bluetooth Devices in Action

beacons

I was recently lucky enough to get a hold of a couple of sets of Bluetooth low-energy beacons; three Estimote beacons and a set of five Kontakt beacons. Whilst the Estimote beacons sure looked a lot fancier, I was keen to see if it was all just for show, or if they truly were a superior product. So I got to work putting them through their paces by conducting some controlled experiments to test their behaviour. Throughout the rest of this post I’m going to discuss the results of these experiments and conclude with some observations on their real-world usefulness for providing proximity-based services.

So What IS a Beacon?

Basically a beacon is a low energy bluetooth device that transmits packets of data that allow smart devices (phones, tablets, computers etc) to be informed when they are in range. Not only are the smart devices capable of being alerted when they are in range, but they are also capable of calculating their proximity to the beacons. The details of this second point are quite scarce and something I’ve been keen to explore further. I wanted to know just how precise this proximity was, and what factors could influence it’s accuracy.

What was I Working With?

As previously mentioned, I had two brands of beacons to play with, Estimote and Kontakt. Both can be purchased from their respective websites. I’m sure you could Google the products, but they’ll likely change over time, so here’s the photos of the exact beacons and packaging that I used in my experiments. I was also using an iPhone 5 running iOS 7.0.4.

Estimote Beacons

Estimote Beacons

Kontakt Beacons

Kontakt Beacons

Under the Hood

The first thing I wanted to do was see what was inside these beacons. So I grabbed a pen and pried one open.

Kontakt Beacon Insides

Kontakt Beacon Insides

Apart from a decent sized battery, and a small circuit board, there wasn’t much more to the Kontakt beacons. The Estimote beacons were a little harder to open as they seemed to be held together with a bucket load of glue. So I left them alone, though it did leave me wondering, how was I supposed to change the battery in two years when it went flat?

The Experiments

I wanted to determine whether I could use these beacons to determine a user’s location with relative precision, within about a 30cm radius of accuracy. I knew that the best way to do this would be to set up a grid of beacons and use some sort of triangulation algorithm to position the user inside the grid. But this all seemed a little too overwhelming to dive straight into. So I thought I’d strip it back, by devising some much smaller experiments and build it up from there. If I could get a better understanding of how the beacons behaved in isolation, I felt I’d have a better chance of designing a system of beacons working together to position a user within the system.

Using the iOS 7′s CoreLocation API is fairly simple, and there are plenty of tutorials on the web for how to get started. I began by writing an app that would simply display a table of ranged beacons, their RSSI (received signal strength indication), accuracy, and proximity values. Then as updates were received from the CLLocationManager, the app would refresh the details of the beacons in the table. This was fine for an entry point app, but anyone who’s used beacons before would know, the numbers fluctuate far too much for your eye to make any judgement of what is being represented. So I decided to modify my app slightly to save all the data and allow it to be exported as a csv, then it could be emailed directly from the iPhone. I could now run some controlled experiments, record all the data, and email it to myself after each experiment. This way I could plot the data and derive some conclusions about the beacons in different scenarios.

Experiment 1 – Testing the Beacons

My first experiment was to just record the beacon’s behaviour at a constant distance of 1.5 meters over the course of a minute. I wanted to look for differences between the two brands, as well as how consistent the beacons were within the same brand. In the following charts I’ve plotted the value of the ‘accuracy’ variable provided by the CoreLocation API. This is a value that I’ll explore in more detail later in this post, but for now all we’d expect to see from a stationary iPhone is a stable accuracy, that’s consistent amongst the same brand. I must say the results were a little surprising.

Estimote Beacons at 1.5m

Kontakt Beacons at 1.5m

The first thing that can be seen is the differences between the two brands. Whilst the Kontakt beacons all hovered around the same accuracy level, the Estimote beacons were far more dispersed. The Estimote beacons varied from a maximum of 6.52 meters down to 1.48 meters, a differential of 5.04 meters. On the other hand, the Kontakt beacons differed in their accuracy by only 1.18 meters. I look further into why this was the case in Experiment 4 below.

Another observation of this experiment was the rate of dropouts between the two brands. The Estimote beacons did not dropout at all, however all the marks on the Kontakt chart along -1 meters are dropouts. This is where the API informs us that it knows it is in range of a beacon, however it has no idea of how far away, and hence no idea of the accuracy.

From this experiment I’d established that the Estimote beacons were somewhat inconsistent between themselves, though in isolation they did seem to be more reliable and have a steadier accuracy. Whilst the Kontakt beacons were a little more spasmodic they did seem to be more consistent amongst themselves.

Experiment 2 – Accuracy vs Distance

There is a fair bit of discussion on the web around the CLBeacon’s ‘accuracy’ property. There is an accepted Stack Overflow answer suggesting that this is just iOS speak for ‘distance’. In the Android port of the iBeacon library the accuracy variable is documented as “A double that is an estimate of how far the iBeacon is away in meters”. However the iPhone SDK documentation clearly states not to use it as a measure of distance from a beacon. This begs the question, why is it in meters? And why does the iPhone SDK documentation say “measured in meters from the beacon”?

Of course this property really has nothing to do with the beacons themselves, and is just each client’s own interpretation of the signal strength received. To test CoreLocation’s implementation of the accuracy property I set up an experiment where I moved a beacon backward by 50cm every minute whilst recording the accuracy. If accuracy is the same as distance, then I’d expect to see some sort of correlation with the actual distance the iPhone is away from the beacon.

Moving Estimote Beacon by 50cm

At first glance it would appear that accuracy does seem to resemble distance. However at 0.5 meters, the accuracy underestimated distance, and by 1.5 meters, accuracy was an overestimation of distance. I decided to extend the distance out to 8 meters to see if the accuracy continued increasing with distance, however over the course of 80 seconds the accuracy of the beacon only reported an accuracy of 3.87m - some 4.13 meters shy of 8 meters. This was less than convincing that accuracy was representing distance.

In certain ranges with this beacon there does appear to be some correlation between accuracy and distance, but is this only coincidental? After all it does makes sense that accuracy would be improved the closer you are to a beacon, and would become worse the further away you are from a beacon. So although some correlation can be seen, can we rely with certainty on accuracy to be used as a measure of distance from a beacon?

Experiment 3 – Power Level

One of the cool features of these beacons is that you can configure some of their properties yourself. By using an iPhone app called BLExplr (available from the app store) you can get in and adjust the configuration of your Kontakt beacons. And by using the Estimote Virtual Beacon app (also in the app store) you can toy around with the Estimote beacons.

I began by doing what all good developers do… Cranking the power level (txPower) up to 100%! With the power level at it’s maximum I took my Kontakt beacons and ran a similar experiment to the previous one, where I would plot the accuracy at different distances. If the power level was higher, assumably the signal strength would be higher too and hence we’d expect a greater accuracy.

Kontakt Full Power

As suspected this appears to be the most convincing evidence, that accuracy has no relationship to distance. Notice that with the Kontakt beacon at 100% power (0dBm), the accuracy hovered just above 0 meters throughout the six minutes of the experiment. If accuracy was distance, you’d expect the calculation to take txPower into account and give us a value that much more closely resembled the actual distance.

To confirm this I decided to run the same experiment but with an Estimote beacon. I turned the power up to 100% (+4bBm) and measured the accuracy at the same varying distances.

Estimote Full Power

Wow! This seem to completely contradict the results seen from the Kontakt beacons. This time I was seeing a very close relationship between accuracy and distance between 1 and 5 meters, though it did start to slip at greater distances (however at 15 meters there were desks and computers obscuring the path, so perhaps this is expected). Why was the iPhone compensating for the increased txPower with the Estimote beacons, but with the Kontakt beacons it was not? Which was the correct interpretation? I decided to dig around further to determine whether accuracy could be trusted to represent distance like the Estimote beacons were showing, or if the Kontakt beacons were correct in showing improved accuracy because we had a higher power.

The answer lay with ‘measured power’. Measured power is the value of measured RSSI at a distance of one meter. It is passed by a beacon in it’s packets to help devices calculate distance based on their current RSSI. It is supposed to be altered by the beacons when txPower is set so that the receiving device knows how to interpret the power of the signal – whether it’s high due to increased power or high due to close proximity. The Estimote beacon knowledge base and the Kontakt beacon documentation claim to set this value in their becaon’s packets, though it appears my Kontakt beacons must have forgotten all about this. Now my iPhone thinks the high signal strength it’s receiving from the Kontakt beacons is as a result of close proximity not due to a high txPower.

So according to both the beacon brands, measured power is provided to help the client (in this case the iPhone) determine the distance of the device. And this is exactly what we are seeing with the Estimote beacons and the iPhone’s accuracy. However we are not seeing this with the Kontakt beacons, leading me to conclude that my Kontakt beacons are in fact faulty. After contacting the manufacturer of these beacons and describing the problem, they were unable to help, pointing out that the use of a third party app (BLExplr) was a potential cause of the problem. This was strange considering that this is the app they instructed me to use when I received the beacons. They did tell me that they will have their own app released soon, so perhaps when this is released, the problem will be solved.

 Experiment 4 – Beacon Inconsistency

As discussed measured power is controlled by your beacons, and cannot be altered by the user. This means that if configured incorrectly you can get vastly different accuracy values from one beacon to the next at the same distance. But are there other potential causes of inconsistency?

I decided to run an experiment where I would test the affects of rotation on my beacons. I would rotate the beacons by 90 degrees every 120 seconds, and if the orientation was playing a part in the beacon’s performance, then I’d expect to see a change in the beacon’s signal strength at different orientations.

Estimote Rotation

It is clear from this chart that the orientation of the beacon was causing a change in the beacon’s signal strength. Notice that between 120 – 240 seconds (at 90 degrees rotation) this beacon’s signal strength was increased, and also much more consistent. To what degree this is changing the accuracy can be seen from the following chart, which is the exact same 480 seconds as the previous chart, but showing the calculated accuracy.

Estimote Rotation Signal Strength

I’d also seen discussion on the web around the impact of closely grouped beacons. Is it possible that if beacons are positioned too close together that they could be interfering with one another? In order to test this, I decided to start recording a beacon’s signal strength whilst the beacon was in isolation. Then after each minute, I’d introduced a new beacon right next to the existing beacon. If interference was occurring I’d expect to see a noticeable change in the original beacon’s signal strength.

Estimote Interference

From this chart, we can see that there is no change in signal strength between the six minutes. In fact, the average signal strengths are -76.34, -75.92, -75.88. We do have a maximum of -72 db moments after the introduction of the final interference beacon, however this is not maintained, and -72 db is actually a stronger signal strength rather a degradation of signal strength. So it can be concluded that with three beacons, no interference is present.

Whilst interference is not playing a part in the beacon’s performance, rotation certainly is. It appears that the internal antenna must be favouring a particular direction, and this is having a major impact on the beacons. In a real world scenario it would be impractical to determine exactly which orientation each beacon should be facing for best performance. Further more, depending on how you are using your beacons you may need them to work through 360 degrees, so positioning of the antenna is irrelevant. This is just something to be aware of.

In the Real World

Having completed these experiments, I’ve been able to conclude that achieving accurate distance measurements with a single beacon is potentially possible, provided you have the right beacon and the right brand. Out of eight beacons, I only found one that was consistently achieving accurate distances. Given that there is so much inconsistency between the beacons and there is no guarantee you will even receive a beacon in an order that does provide accurate results, it seems the only way to achieve accuracy is by using many beacons and averaging the results.

So is it possible that by using multiple beacons each providing somewhat inaccurate data that we can deduce something more accurate? I’ve spent some time trying to achieve this, so please stay tuned for my upcoming blog post where I’ll discuss the results of these additional experiments, which involve tracking a moving iPhone through a grid of tactically placed beacons. I’ll also take a more in depth look at using beacons in the real world but now with a better understanding of how beacon signal strength, accuracy, and proximity should be interpreted.

This entry was posted in iPhone and tagged , . Bookmark the permalink.

9 Responses to The Beacon Experiments: Low-Energy Bluetooth Devices in Action

  1. Nice Post!

    I’m currently involved in an iBeacon Proof-of-Concept for a big box european retailer, and the most significant project-item regards the shopping-experience.
    Engaging shoppers too much means disturbing them, so you have to determine the optimal “interaction” trade-off.

    I published a white-paper about iBeacons-at-work titled “iBeacon Bible”; feel free to download it from my blogsite http://www.gaia-matrix.com; it can be valuable if you’re involved in this technology or just want to know more.

    Andy Cavallini

  2. Thank you for helping me avoid days and days of sorrow trying to figure this stuff out (or, rather, pointing out that it’s not feasible for a typical app developer to figure out). Looking forward to next post!

  3. Hi there,
    This is Wojtek Borowicz, I’m a community evangelist at Estimote.

    First of all: that’s an amazing piece of work. Great job, and cool tests, I’m sure it will be pretty informative for developers and people experimenting with Beacons. But as for calculating distance, please keep in mind that iBeacon standard as a solution isn’t designed for accurate location, at least at this point. The main aim is to provide users with proximity-based context.

    It is of course possible, to build good location services, but it’s really complex from the programming point of view.

    If you want to know something more about Estimote, feel free to just drop me a line: wojciech.borowicz [at] estimote.

    Cheers.

    • Wojtek… the problem of course is that in many environments you need to know which beacon you are closer to. To do it well *is* hard, but that seems like a great opportunity for you guys to differentiate… instead of forcing every app developer to figure this out, you figure it out and put it in the SDK. Otherwise we’ll continue to see lots of POCs and very few deployments.

  4. Of course you are right, but keep in mind it’s not only the case of knowing which Beacon you are closer to. At the end of the day, all Beacons are just small radio transmitters, and radio waves are suspectible to many sources of interference. In many cases, especially in indoor environment, you also have to include things like multipath or wave diffraction. It all comes together to create noise, and decrease the positioning accuracy.

    Of course, as the tech grows more mature, both on the software and hardware side, those problems will be addressed. We’re working hard on this at Estimote, as I’m sure other vendors are doing as well. 3rd parties and developers are also tinkering around that, so we’re headed in the right direction. But there’s still a lot of distance to cover :)

    Cheers.

  5. Great article Chaise! Those experiments and analysis are an excellent resource.

    One thing I’ve found when experimenting with beacons (I’ve only tested Estimote ones), is that the more you move your device the more accurate the measurement reading is. It sounds strange but just moving the device (ie, iPhone) rather than having it sitting on a desk appeared to produce more accurate and consistent distance measurements. Possibly it is to do with tilting/direction more than moving its physical location.

    It might be interesting to see one of your experiments with that concept as a focus.
    Looking forward to your next post!

  6. Ron says:

    Hi, great article. I wonder if the problem is more to do with “calibration” of the devices – that is, for each device, a calibration needs to be performed and the formula for each calibration imprinted on the device. That is, for every device you put in the field, measure RSSI at 1 metre and set this as the calibration constant for that specific device. Then when a device is picked up, the constant is grabbed from a server that stores this stuff, then use that for distance measure?

    • Hi Ron,

      Calibrating the so called Measured Power (RSSI value at 1 meter distance) of course is important for proximity measurements, but it still won’t bend the laws of physics for you :) Even well calibrated, radio signal is still susceptible to numerous factors, that can easily alter the readings. Hence sophisticated noise cancelling is a must if what you want to achieve are really accurate results (which, on the other hand, isn’t really necessary for most use cases).

      Cheers.

  7. > Out of eight beacons, I only found one that was consistently achieving accurate distances.

    Curious as to which device this is.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s