Sufficiently Advanced Design


Thinking out loud

Posts tagged troubleshooting
Trouble: shot, or how I learned to stop worrying and love AC

The short version: everyone was right. Induction caused massive AC on the sensor data wire.


I added a makeshift low-pass power filter with a resistor and capacitor I had handy in my tool bin, and it solved the sensor issue.

Turns out, 25 meters of narrow gauge wire is a terrific antenna.

I need to do a proper measurement of the AC frequency on the power rail and design an RC filter to match it for the production design, but this is good enough for now.

Witch Lights troubleshooting, this sensor is going to kill me edition

I think I figured out why this motion sensor keeps going haywire.

(TL;DR: The more lifelike trail animation causes low-frequency AC and induction on the long power harness. I think.)

Witch Lights 2's sensor2worked just fine in July 2017, with version 1 of the FastLED implementation, which brought multi-sprite automation, and the SpriteManager and SpriteVector framework to manage and create/store/delete Sprite objects in memory, respectively.

It only started malfunctioning in late June 2018, when I pulled the harness out of storage in the spare bedroom for the winter.

What changed?

This year, just the software.

The v.1 FastLED Witch Lights were completed at the last second: the last commit date on GitHub for that build was submitted after midnight on July 2nd. The priority had been fixing memory leaks: the Witch Lights run for 6-9 hours on a single battery in warm weather, and a slow memory leak could cause them to crash in hour 4.

Fortunately, Jim is an excellent developer, and did in fact build an object-oriented animation framework for the Witch Lights gratis, so I owe him a perpetual debt of gratitude. (He also walked me through the class inheritance and the way the state machine sprites were configured and such when I began work on v.2 in spring of 2018. He's good people.)

Anyway, the way we drew the sprite at the time was, we loaded a static CRGB struct into RAM, and then used memcpy to update the leds struct (also CRGB which is like an array of 3-byte arrays, each containing an R, G, and B value) with a function called strpcpy.

(We did it that way because we draw the sprites 3 pixels off-strip, and having an abstracted function that we could feed a location of -3 without accidentally overwriting the bootloader—again—helped a lot.)

Anyway, we just moved pre-rendered sprites around for the travel mode of the Sprite state machine. The idle animation of the time also was a set of pre-rendered animations loaded into RAM as series of CRGB structs that got stepped through like animation frames.

We did that because this was the first time we were going to try animating more than one sprite at a time. We knew the Arduino Due was a significant upgrade from the Duemilanove the first builds of the software was tested on. We didn't, however, have a feel for what the processor capabilities were.

We only knew that v.1 ran for hours without crashing in 1K of SRAM while running a 30-pixel test strip on the 8-bit Duemilanove, and the Due had 96K to run 600-750 pixels. So we threw RAM at the first draft implementation.

insert animation comparison video

In the first half of this (over-long, but forgive me, it was done in 30 minutes before leaving for a trip) documentation video, you can see the result.

Here's a detail.

Witch Lights test home.gif

Note that the length of the tail is exactly the same throughout its travel. It's a series of pre-rendered color values. So, even though the sprite starts off slowly and accelerates, the trail doesn't stretch to emphasize that naturalistic motion.

In version 2, I fixed that.

Turns out, the ARM chip in the Due provided us with excess processor capacity, and so I could afford to play around a bit. Instead of a long, pre-rendered set of fading pixels, each traveling sprite is now three pixels long. And when a sprite moves forwards one pixel, it calls a recursive DimTrail() method on the pixel it left shining behind itself.

DimTrail() uses the FastLED FadeToBlackBy function to fade the brightness of its pixel by a factor which scales with the current updateInterval (framerate) of the Sprite. In english, it fades a certain pixel more if the Sprite is moving slowly, and fades less the faster the Sprite moves.

And then it calls DimTrail() on the pixel behind it.

(There's logic to prevent the Sprite from eating recursion and dying, don't worry.)

As a result, the sprites now have stretchy, lifelike trails.

Witch Lights CO dark.gif

And that means that the total number of pixels we're powering changes unpredictably. And it stands to reason that the amount of current the NeoPixel strips pull from the power rail will also change unpredictably.

In english, the changing trail length creates alternating current (AC) on the power rail.

The same power rail that is powering sensor2.

The 85-foot-long power rail.

That one.

Everyone was telling me it sounded like I needed a low-pass filter on the power line for the sensor at the end of such a long power harness. And I agreed. But what was killing me was:

Why did it work in 2017, and not now? What changed?

The software changed.

I also didn't understand why sensor2 would seem to work fine for some random but short amount of time, and then malfunction. It's because, if there's nothing animating with DimTrail(), we don't have AC on the power rail. As soon as we get one or two sprites animating for a few seconds, AC along the 85-foot power lines build up, and sensor2 probably does not like that.

Several friends with more (real) electronics experience said it sounded like AC or inductance on the power pin was making the sensor behave unpredictably.

The Adafruit Discord Server suggested testing an RC filter. They also believed low-frequency AC on the power pin would definitely cause these symptoms. As did I.

I just didn't know why.

I still don't. Not yet. I have to look for the thing I've predicted will be there: AC on the power line.

My multimeter isn't the right tool to find this bug. Fortunately, a generous friend has mailed me an oscilloscope, and it arrives Wednesday.

Once I have that, I can measure the frequency of the oscillation on the power pin (assuming it exists), and design an RC filter, and fix this without installing a separate power supply for the sensor lines.