Simple data filtering

I looked through some of yesterday's data using GNU Octave with a goal of understanding how digital filtering can be used to smooth it a bit. I've been thinking of Kalman filters, but am realizing that I really don't have a predictive model of my system -- it's not like I can do some sort of dead reckoning from my current state to a new one. So for now I'm considering simple lowpass filters.

The main tradeoff from my viewpoint is the phase delay of the filters -- roughly speaking, the more smoothing, the more delay in the response of the smoothed output. This is rather bad for an instrument that attempts to give the pilot an "instant" view of their airdata.

On the other hand, I do think, from my earlier tuft experiments, that the airflow in my road testing is really horribly turbulent. I look forward to flight tests where I can quantify what things really look like off an airplane wing.

For now, I'm going with the assumption that if I can do a little bit of filtering to eliminate much of the jitter, that will be good preparation for any future contingency. And it's easy to implement a filter on the display side, and just as easy to disable it if I find I don't need it.

With that in mind, I used Octave's FIR filter design tools to come up with a grid of filtering options, with filter order ranging from N = 5 to 40 taps, and lowpass cutoff frequency ranging from f = 0.4 to 1.0 Hz. Here are the time domain results. Note that this is a very high-resolution image -- you can click on it and zoom as you wish.

The striking thing is that the most important criterion is filter order, not cutoff frequency. That has the most effect on time domain qualitative "smoothness" and phase delay. And it appears that at N = 5, our delay is less than 1/4 of a second or so, which seems reasonable for an aircraft instrument.

On this basis, I think a simple digital filter in the display along the lines of the above is a good idea, and I'll be adding it and reporting back as to the results.

Data from probe on a stick

We did a road test with the POaS today:

This is the set of all data traces for one "loop", showing reasonable data quality:

Now a couple of interesting things.

First, I know that my peak speed on this run was 75 mph. Converted to a dynamic pressure, this would be 674 Pa. Here is a zoomed-in region of the delta P 0 measurement at the peak:

Lots of noise, but the data is believably close to the right value.

Second, there still are some single-sample "spikes" as visible in this zoomed-in region of delta P 0. This is specifically visible when the readings are low -- meaning that these spikes are otherwise dwarfed by other sources of noise:

Note that the baseline reading is only 4 Pa, and the spikes go down to -35 Pa. These are pretty low numbers overall. My suspicion is that these are either vibrations of the car, or actual noise. In other words, at about 20 Hz, the pressure sensors are acting as microphones and picking up things like the car doors slamming. This is worth testing more systematically at some point in the future.

Update: Looking at the last plot, these are not single-sample spikes! Many of them span multiple samples, meaning they are much more likely due to some physical phenomenon and not just noise in the digital lines in the probe. What phenomenon exactly -- and why the spikes occur exclusively in the negative-pressure direction rather than positive -- is a question for the ages.

Probe on a stick is done

The PO[a]S is done and sending data. Stay tuned for road tests!

Probe on a stick

Given the issues with the PCBs, I'm building a really simple "probe on a stick" that I can use to take data and de-risk other portions of the project. I don't think this is a valid flight test article but at least it can allow me to try road tests at various angles and see what results I get, and experiment with digital filtering of my signals.

On some level, the road tests did their job -- they acted as enough of a shakeout of the probe that they uncovered underlying reliability problems without the expense of flight testing. But it sure is a bitter pill to swallow....

I get exactly 5 minutes of crying in the corner ... but then I must press onward regardless.

A hypothesis for the gremlins

My friend Jeremy Cole (https://blog.jcole.us/) who is a very qualified electronics hobbyist (and who does not live locally, so we did this remotely via pictures) has a hypothesis about my board problems.

We probed the bad boards and found that they were just shorted VCC - GND. Not a low resistance or anything -- just a plain short. Yikes.

After much diagnosis, he thinks the soldermask on my boards is misaligned, which caused the first dot of solder to short VCC to the ground plane. Here is a zoomed-in view of one of the bad boards. You can see the offset soldermask in the vias, and the suspicious area in capacitor C2 that he called out:

It is not a big stretch to imagine that this is the source of my "spikes" due to vibration on my otherwise "good" board too -- if something is really, really close, then it could indeed have an intermittent short.

I am very grateful for my friend's help, without which I would have wallowed for many weeks or months without resolution. I'm taking a day off the Airball train to hang out with my family but will resume work tomorrow and post more updates as they come in.

More PCB manufacture gremlins

I was talking to a friend last night about how I'm using a reflow tool that another friend kindly lent me. He asked me what temperature I was using. "400" I replied. 400 ... what? Celsius? Fahrenheit? That led to some hilarity where it became clear I was frying my chips with 400 Celsius hot air then expecting them to work.

Today I set myself the task of creating  new board from scratch. I tried the recommended 235 Celsius or thereabouts but I was having trouble getting the solder to reflow, so I went back to soldering with the iron. I got to this point:

I then plugged in a Fio V3 and verified that the I2C mux was working, and that the temperature sensor chip was showing up correctly on bus 0x04 at address 0x48. All seemed great.

I then went back and installed the IC headers for my pressure sensors. Same workflow as yesterday. After that step, I had the following:

I then plugged in the Fio V3 and ... silence. Nada. Nothing.

Could it be that I fried the I2C mux? I de-soldered it and installed a new one. Nope. Didn't help.

Could it be that I needed to wash off the flux? (Grasping at straws here.) No, didn't help either.

Could the Fio V3 be busted? No, when removed from my board, it works just fine.

Could the Fio V3 be busted in some way that removing it from my board does not detect? No, when I plugged it into my working-but-slightly-flakey board that is populated with pressure sensors and that I used for my road test yesterday, it worked fine and sent back pressure signals.

My only conclusion is that I have no ability at this point to reliably manufacture the simple circuit I have designed.

Strangely unsuccessful PCB fabrication

Given that my next step was to build a new board and hope the flakey gremlins went away, I put myself to work.

I used some solder paste and a reflow hot air gun that a friend had very kindly lent me, and it simplified the work quite a bit.

I got to the point where the I2C mux and the temperature sensor on the board were working just fine, and I had a socket soldered down for a Fio V3.

I then soldered on some IC sockets for the pressure sensors. And all of a sudden, that by itself caused my board to stop functioning.

I'm not sure what went wrong but it seems that the Fio V3 freezes when writing to the I2C mux somewhere. I scoped the signals and don't have an immediate diagnosis. But it's very frustrating that I can't seem to put together a board without high drama. :(

These are pictures of both sides of the board as it is. As you can see, the 4-pin IC sockets are pretty far away from anything, so I remain puzzled as to what evils soldering them on could possibly have wrought.

Partially promising road run

I used some rubber bands to make sure the pressure sensor ICs were secure in their sockets, then went out for a quick road run. The following is the data I acquired.

The first is all my data from the entire run. Note what looks like a nice, stable temperature reading, and pressure readings that seem stable at first but start displaying "fuzzy" spikes as we get going:

Zooming in on one area, note the spikes in the delta_P_0 and delta_P_alpha signals in particular:

Note how the spikiness showed up after some jostling on the road. My conclusion is as before -- that I've licked some of the problems with software, but there must linger some more issues with the hardware that are causing noise in my readings.

My best recourse at this point is to simply build up an independent board, install the same sensors in it, and give it a try to see if it has the same behavior. If it does, then that warrants going in there with the analog scope to see what is going on with the signals. If not, then I'll just consider this an artifact of poor manufacturing. Off to the soldering station with me!

Most (not all) noisy readings are software problems

I discovered that I had a problem in my software where I was using Arduino timers incorrectly, which is what caused most of my random "spike" problems. This is what my data looks like with this fixed. There is still a "spike" or two and I suspect poor board manufacturing -- the same thing that causes my chips to freeze up -- but I need to verify that. Again, this is the sensor just sitting on my bench:

Probe mount for the Dynamixel servos

I made a tiny mounting plate to attach the probe to the Dynamixel servos. This would be a relatively prosaic exercise were it not for the fact that I am trying out OnShape -- a cloud CAD system reportedly built by some engineers from SolidWorks. So far I find it pretty useable, though different enough from SolidWorks to require some thinking.

Of course the perennial problem is how to get a full-scale drawing printed out with the proper dimensions. Just hitting Print in the browser won't cut it. I finally figured out a workflow that works -- exporting a PDF from OnShape, then opening it in Preview.app on a Macintosh, and printing to our HP inkjet printer from there. Other workflows may work, and conversely, this one may not work for you, but there you go.

Now with that done, it appears the Dynamixel servos may not be rigid enough, after all, for the purpose I have in mind. They will likely wobble quite a bit when mounted on top of the car, and there's a lot of lateral free play in their bearings. I'll get my control board in sometime next week and power them up and see, and then make my decisions. There are larger (but alack, more expensive) Dynamixels out there, and there's also the option of using something like stepper motors (alack, more work).

Possible noisy I2C or power lines

This is not a progress report regarding Airball per se, but rather a discussion of the kinds of problems one obtains when doing homebrew PCB design and assembly without much experience.

My last post noted how apparent transients were causing the pressure sensors to freeze up, and they would thereafter not recover without removing the battery. Here I'm going to show another effect of what may be the same underlying problem: unexplained blips in the sensor readings. Note that my probe reports data in engineering units, so these are not the binary results. But the results I observe lead me to suspect noise of some sort leading to incorrect readings over I2C. Here are the 4 variables -- temperature and 3 pressures -- reported with the probe sitting on my desk:

And here is a zoomed-in chunk of the delta P beta reading showing just one data point of the jitter:

Clearly that's no way to take data. :/ Also note that while the temperature signal has some jitter, it has less of it than the pressure sensor signals.

My suspicion right now is that something is deeply wrong with my board. A trace is broken, or a solder joint is not complete, or ... something. So there are two avenues along which to proceed:

1. Build up a new board and see if that fixes things; and
2. Scope the I2C signals and see if I can detect undue noise.

Sensor flakiness most likely loose connection and vibrations

Recall how the probe in this morning's testing was cantilevered out at the end of a skinny aluminum beam, allowing it to wave softly back and forth?

Well this afternoon I ran some tests with the mast extended, and therefore more rigidly supported in the vertical direction. Boom, sensor lockup. I added some logging to my probe protocol to send me a message when it is "waiting" for a sensor to return a reading, and sure enough:

0.00,294.90,52.86,1.92,9.41
# .
# .

The lines starting with "#" are log lines, and what you see is first a data line (the comma separated numbers) followed by a couple of log lines (which continue on and on) that say the CPU is waiting for a reading from one of the chips.

Again, the way to reset is to disconnect the battery, wait a while, then reconnect the battery. Hm.

I went out again and noticed that, as I went over a bump in the road, the sensor hiccuped a little. Then it got back on track.

AHA.

I added some more logging to find out which sensor was freezing, and now when it froze my log lines always looked like this:

# . 0x02 @ 0x29

Ah. This means the failure was always on I2C bus 0x02, address 0x29. This just happens to be one particular location on my board.

I took the probe to the bench and applied vibrations with a motor, and I could reliably reproduce this exact failure with this same chip.

Now of course the question is: what if I swap chips in location (bus 0x02, address 0x29): Does the problem still occur, and if so, is it still at the same location? Well let's find out. I swapped the chips in locations 0x02 and 0x03 and tried again. And lo and behold, after some vibration, a whole string of:

# . 0x02 @ 0x29

That's pretty strong evidence of some problem -- possibly a bad solder joint -- in the board in that location.

So we now know we have the following problems:

1. Board is not vibration safe. Most likely a loose connection, or perhaps a bad IC socket, along with vibrations, creates a flakey connection.

2. Sensor chip freezes upon flakey connection. As a result of the flakey connection, the sensor chip freezes and needs to be power cycled to reset it.

3. Fio V3 battery power circuitry never truly shuts off. As previously documented, the circuit maintains a nonzero VCC even when turned "off", so the battery needs to be disconnected, and the capacitors allowed to discharge for a while, before the sensor chip can power down enough to reset.

Solutions I could propose would be:

1. Pay more attention to board manufacturing and qualify boards with vibration testing before investing in an expensive flight test. Ultimately, maybe get professionals to assemble the boards.

2. This is not something I can control -- it seems to be a feature of the All Sensors chips. But I have 3 Honeywell chips on order and I will see how these perform under similar conditions.

3. I can add an extra switch in the probe for now to disconnect the battery -- I had that in an earlier version of the probe and thought I didn't need it any more, but I was wrong :), and it's easy to wire up.

Overall, now that the problem seems to be isolated, I feel like the project is back on track.

Dynamixel robot actuators

After my poor experience with hobby servos, a friend recommended Dynamixel robot actuators. Their entire reason for existence is to accurately go to a given absolute angle in response to commands over a simple, convenient network protocol (in this case, a half duplex daisy chained 3-wire connection that also supplies power).

I got two of the most popular model, the AX-12A:

As you can see, the servos come with enough mounting hardware included to build a pan/tilt mount. (The hardware is available as open stock, allowing you to mix and match and create a whole robot in Lego-like fashion.)

I'm waiting on shipment of a kit that includes the USB2AX interface dongle. The USB2AX seems weirdly overpriced for what it is, but it sure beats having to mess around trying to get a generic full duplex serial interface to work in half duplex mode, or bit-banging directly.

I will report on results one way or the other.

Sensor flakiness is flakey

By definition, a flakey sensor is flakey in its flakiness right?

First thing this morning, I set up the sensor on a brick in my workshop and blew air on it with a fan for a while. It seemed fine.

Then I set up the sensor on my mast in its retracted position, which (I hypothesize...) does not give as clean airflow but is good enough for reliability testing. Melissa and I drove around town with it turned on.

Note how the mount does not quite allow it to be set up horizontally, but it simulates about 30 degrees "angle of attack" in this position which is just fine for getting reliability data. This is Melissa acting as PIC of the test vehicle.

The result was that, after a whole morning of shopping and driving around, the probe worked just fine.

Well schmell. That's frustrating. Not only an intermittent failure, but one that's dastardly hard to reproduce.

The good side is that we can continue to acquire data and work around this problem with ops -- e.g. I suspect that disconnecting the battery after switching the circuit off drains it down enough to reset whatever is in there that gets confused. One version of this probe had an on/off switch for the battery itself, which I can easily resurrect. Also, I need to take more data on the voltage drain-down.

Flakey sensor or circuitry again

After gaining confidence that, with proper ESD protection, the probe circuitry was not destroying my sensors (any more), I installed 2 more sensors for a total of 3 (alpha, beta and dynamic pressure) and went out for a road test.

After a few minutes' worth of run time -- some of it when parked and raising the mast, and some at the beginning of my road route -- the probe stopped sending data. It resumed momentarily after a minute or so, then stopped again and did not restart.

I had my computer handy so I did some debugging, and it turns out that one of the sensors was freezing up. It enumerated properly on the I2C bus, but when asked to give a pressure reading, it hung in its "waiting" state forever. The other two sensors worked fine.

Power cycling the probe did not help immediately. Note, from previous work, that the probe remains powered up to some small voltage due to the battery, and I did not remove the battery.

Later on, after leaving the probe off for a while and turning it back on, it started working again magically. Note again from previous work, that when turned off with the battery connected, the probe VCC dies down to a small amount over a few minutes, then remains at that small amount thereafter.

My current provisional conclusion is that the sensors are flakey, either because of how I am using them, or some detail of my circuit, or some internal issue that I am running into. I do not currently have an obvious way forward apart from an extensive re-design of my electronics and/or extensive debugging of the problem hoping to reproduce it.

Hobby servo accuracy is poor

I have been thinking of building a pan/tilt mount for the probe, at least for the road testing to allow me to program "sweeps" through different AoA/yaw combinations. Pan/tilt servo mounts are pretty common, using hardware like this one, so I figured I'd give them a try.

I purchased a Hitec D645MW servo and a Pololu 6-channel Micro Mastro USB controller and tried them out. Using the published conversion factors for the servo, I tried commanding it to 30 degrees and 60 degrees, based on the book conversion between PWM pulse width and output angle (0.08 degrees per microsecond). Here are the results below. Clearly the conclusion is that "out of box" accuracy for hobby servos, even this relatively nice one, is not to be expected.

I'm upgrading to Dynamixel AX-12A actuators and will publish my results.

Poor filtering is poor

I noodled around with the data from the road test, passing it though a 25-tap lowpass FIR filter. Here are the results -- purple is original, and green is filtered. The X axis is in seconds:

Looking more closely, we have:

Notice the 1 second delay, and the DC bias. Clearly this is not okay.

Now it's true that I studied signal processing at some point in my life, but that was literally 30 years ago. :) So I have a big ramp-up I need to do, and clearly I need to learn about, and implement, a proper Kalman filter.

The data above is nominally at 20 Hz, but because I have had trouble getting timers to work on my Arduino, what this really means is that I acquire data with a 50 millisecond delay in between. This is no way to acquire data. :)

And finally, it's possible that the road environment has more turbulence than flight, so it may be that the filtering requirements for flight are less stringent, which would mean I could defer the fancy filtering until later.

Therefore what I've learned here is:

1. I need to move to flight testing sooner;
2. This means I need to figure out my airplane mounting again -- the mount I bought was a bit wobbly, and maybe I should move to using a Citabria that has 2 struts for more stability;
3. For this testing I need to buckle up and add my 2 other sensors to my probe and hope I don't fry them;
4. I also need to get Arduino timers working so my data is at a consistent sampling rate (no sense paying for a flight test then using crappy sampling); and
5. Meanwhile, I need to start learning about Kalman filtering.

That should keep me busy.

First road test, Pitot only

Today I ran my first road test of the actual probe, using only one sensor, the gage sensor for the center hole relative to the static pressure probe. In that configuration, it was just a fancy Pitot tube. Here is the sensor mounted on the mast:

And here is the Airball display unit in the car, along with an XBee receiver connected via USB to my laptop acquiring the data for analysis:

I drove the car while my wife rode shotgun acting as Test Range Safety Officer [tm]. Her job was to look out for stupid stuff, and indeed she did catch some possible snafus with low-lying telephone wires and trees, and kept situational awareness on the general road environment.

Here is a video of a portion of our trip:

We set the cruise control for 80 mph and read off approximately 66 kias on the probe. We are at sea level and as you can see we were at approximately 64 degrees F or about 17.7 degrees C so conditions were close to standard. 80 mph means we should be reading 69.5 kias, which means our error is more or less acceptable for now. The data is not filtered, and the reading was estimated simply by changing the IAS scaling until the ball more or less filled the screen then reading off full scale, so there's lots of error there.

The following is the plot of Q (in Pascals) versus sample number. Our sample rate is approximately 20 Hz -- approximate because I'm still trying to figure out how to get timer interrupts to work on the Arduino.

Here is a 40-second expanded section:

And this shows a bit more clearly what is noise (due to turbulence -- the sensor by itself on the bench is very quiet to within +/- 1 Pascal) and what is speed variation as we navigate the busy Bay Area traffic.

As before, my expectation is that we need some digital filtering.

The probe survived the experience and, so far (knock on wood), with good anti-ESD practices and clean power-down and discharging of the electronics prior to making any changes to the circuit, the parts have survived. If this continues, I will be encouraged to commit more of my precious, expensive pressure sensors to testing. For now, I remain with one sensor in the "fancy Pitot tube" mode.

Water manometer verification of sensor

I built up a new sensor board and, after a whole lot of pain with previous sensor damage, I am taking it slowly. I only installed one of the pressure sensors: the gage sensor for measuring dynamic pressure in the center hole of the probe.

To figure out whether the readings it's giving me are even remotely related to reality, I made a very simple water manometer and connected it to my sensor:

Meanwhile, I set up my PC with an XBee to receive the serial data:

I then applied a measured water column:

and 4.94 in. H2O should be 1229 Pa. My sensor board reports its data in SI units, so looking at the numbers coming off it:

the reported 1224 Pa is close enough for me. This was not an attempt to calibrate the sensor, or to determine its accuracy, but rather simply to figure out if I had a serious math error or mis-configuration somewhere. We are good to go now.

Next up, installing the sensor on the Toyota mast and getting some "flight" data. As I mentioned before I'm probably going to have to make a foray in to digital filtering. Expect more soon.