Sensor flakiness most likely loose connection and vibrations

Recall how the probe in this morning's testing was cantilevered out at the end of a skinny aluminum beam, allowing it to wave softly back and forth?

Well this afternoon I ran some tests with the mast extended, and therefore more rigidly supported in the vertical direction. Boom, sensor lockup. I added some logging to my probe protocol to send me a message when it is "waiting" for a sensor to return a reading, and sure enough:

0.00,294.90,52.86,1.92,9.41
# .
# .

The lines starting with "#" are log lines, and what you see is first a data line (the comma separated numbers) followed by a couple of log lines (which continue on and on) that say the CPU is waiting for a reading from one of the chips.

Again, the way to reset is to disconnect the battery, wait a while, then reconnect the battery. Hm.

I went out again and noticed that, as I went over a bump in the road, the sensor hiccuped a little. Then it got back on track.

AHA.

I added some more logging to find out which sensor was freezing, and now when it froze my log lines always looked like this:

# . 0x02 @ 0x29

Ah. This means the failure was always on I2C bus 0x02, address 0x29. This just happens to be one particular location on my board.

I took the probe to the bench and applied vibrations with a motor, and I could reliably reproduce this exact failure with this same chip.

Now of course the question is: what if I swap chips in location (bus 0x02, address 0x29): Does the problem still occur, and if so, is it still at the same location? Well let's find out. I swapped the chips in locations 0x02 and 0x03 and tried again. And lo and behold, after some vibration, a whole string of:

# . 0x02 @ 0x29

That's pretty strong evidence of some problem -- possibly a bad solder joint -- in the board in that location.

So we now know we have the following problems:

1. Board is not vibration safe. Most likely a loose connection, or perhaps a bad IC socket, along with vibrations, creates a flakey connection.

2. Sensor chip freezes upon flakey connection. As a result of the flakey connection, the sensor chip freezes and needs to be power cycled to reset it.

3. Fio V3 battery power circuitry never truly shuts off. As previously documented, the circuit maintains a nonzero VCC even when turned "off", so the battery needs to be disconnected, and the capacitors allowed to discharge for a while, before the sensor chip can power down enough to reset.

Solutions I could propose would be:

1. Pay more attention to board manufacturing and qualify boards with vibration testing before investing in an expensive flight test. Ultimately, maybe get professionals to assemble the boards.

2. This is not something I can control -- it seems to be a feature of the All Sensors chips. But I have 3 Honeywell chips on order and I will see how these perform under similar conditions.

3. I can add an extra switch in the probe for now to disconnect the battery -- I had that in an earlier version of the probe and thought I didn't need it any more, but I was wrong :), and it's easy to wire up.

Overall, now that the problem seems to be isolated, I feel like the project is back on track.