Kasli noise (3V3MP, P12V0 & Ground)

[pathfinder49]

There is a 35 mVp-p, ~200 kHz oscillation in the 3V3MP supply. I suspect this noise to originate from Kasli.

This noise appears to couple into a lot of other supply voltages and causes several spurs in the output channels.

The coupling to the output channels appears to be dependent on grounding.

Here are some spur measurements for a variety of power supplies and output channels 24-31. The power supplies are labeled in the legend. The DAC outputs are the bottom grouping of traces with the lowest (spur free) measurement representing the noise floor.

[hartytp]

Interesting. The best test for that might be to try removing 3V3_MP from the ribbon cable and injecting an external 3V3MP from a bench PSU. If that removes the output spurs then we should look into filtering that on Fastino.

[pathfinder49]

The previous plot was measured with no additional grounding attached to the board.

Attaching an additional ground to TP17, reduces the amplitude of the output spurs from the 3V3MP supply.

The plot below shows the channel noise with this additional GND attached. The new traces around zero are the is the log ratio of the noise before and after. Peaks reducing in amplitude are indicated by a negative spike here.

[hartytp]

Attaching an additional ground to TP17, reduces the amplitude of the output spurs from the 3V3MP supply.

I don't understand what you did here...

[pathfinder49]

I measured the channel outputs as before. The only difference is that a connection to (building) ground was clipped onto TP17. TP17 connects to the board ground.

[pathfinder49]

The trace at the top indicates the spur difference due to this GND connection.

[hartytp]

Sounds like common-mode noise coming from Kasli. Can you try powering Fastino from an external 12V and 3V3MP supply?

[pathfinder49]

I've powered Fastino from an external supply and disconnected it from other devices.

Here a resulting noise spectrum. (Not entirely comparable as measured in a different location)

[pathfinder49]

Looking at the external 3v3 supply there is 500 kHz saw-tooth like noise. This would explain the most significant spurs.

[hartytp]

Here a resulting noise spectrum. (Not entirely comparable as measured in a different location)

Cool! That's starting to look quite clean. Nice detective work!

So...to recap:

• you've removed some large spurs by stabilizing the N12V0 LDO
• you've removed more spurs by powering from an external LDO
• the remaining spurs at ~450kHz and harmonics are from a TBD source, quite possibly the +-13V SMPS
• the noise floor of that measurement (measurement BW) is comparable to the measurements that @jordens posted previously
• you don't see any channels being significantly worse than that?

If so, great!

I suppose the next thing is to do something like:

• use a scalpel to remove the P3V3MP wire from the ribbon cable
• use super glue / epoxy (hot glue isn't mechanically that great IME) to add some kind of connector to the board (e.g. one of those Molex KK series 2-pins, as the back side is fairly well-suited to glueing IME)
• crimp the 3V3MP wire onto a connector
• dead bug an RC/LC filter onto the connector and then wire the filter output to the ribbon cable connector
• add a wire to connect the filter output to the back of the header/ a 3V3 TP or any other suitable location
• verify the spurs are still gone

If so, we can easily fix this in the next revision by adding a filter.

[hartytp]

AFAICT the 3V3MP current draw is low (a few mA?) so a 100R || 1uF filter should do a good job here.

[dtcallcock]

There is a 35 mVp-p, ~200 kHz oscillation in the 3V3MP supply. I suspect this noise to originate from Kasli.

Worth adding additional decoupling caps (between the fuse and the EEM connector) on Kasli v2 and Kasli-SOC whilst we still have the chance?

[gkasprow]

P3V3MP is used only to supply I2C memory, thermostat, and some logics. It's better to add a blocking cap at the EEM boards. There can also be ripples coming from ground bouncing or 12V return current.

[gkasprow]

Before you claim any oscillations/ripples, first short probe tip to the place you connected the crocodile :)

[pathfinder49]

I do not observe the oscillation when grounding the shield from IC3 pin 3, so shouldn't be relevant.

[pathfinder49]

I had missed the P12V0 rail in my earlier measurments. The noise here is similar to the 3V3MP noise. I will investigate the effect of filtering either/both of these.

[pathfinder49]

I've added a 16.2 kHz LC and 1.6 kHz RC filters onto P12V0 and P3V3MP respectively. This cleans up the respective board voltages nicely. However, this filtering does not replicate the behavior when disconnected from Kasli and powered from an external supply (see below).

Specra measured on ch31

For comparison powering Fastino from an external supply, without connection to Kasli (also ch31).

[pathfinder49]

A preliminary look on a scope suggests the I2C lanes may be coupling to the outputs. (The only EEM lanes with ~250 kHz features). I'll Investigate more properly tomorrow.

[hartytp]

@gkasprow as @pathfinder49 will write up shortly, the remaining spurs are a ground-loop involving Kasli and our spectrum analyser.

There are a few ways this can be resolved, but it got me thinking more generally about EEM grounding. Specifically: is it necessary/correct to tie all the ground pins from the EEM cable to the EEM circuit ground? Are the grounds separating the LVDS lines really grounds, or more like screening? If the latter, shouldn't they be only connected at the source end? Maybe ground these via DNPd 0R resistors so we can change later?

[gkasprow]

These EEM GNDs serve two purposes: they define differential impedance and provide lane screening; they are also 12V return path.
The DC/DC converter is quite well decoupled from EEM connector, I added LC filter also at its input to make sure it won't propagate the current spikes to the inside of the chassis.
It's worth supplying Kasli from linear bench PSU to make sure the 12V SMPS is not an issue here.
Another way of killing the ground loops is using magnetics. You can easily buy IDC tape ferrites that increase the ground loop impedance for higher frequencies.
I wouldn't touch individual EEM cable grounds.

Fastino has the output CMCs disabled by default, It would be worth adding them at least on some channels. Make sure that DC is not passing through the return current because just milliamps saturate the core. One can also add ferrite core at the coax cable output. Take into account that all these ferrites work only for AC and it is extremely easy to saturate them if the return DC current is not passing the same way back through the core. In such case, they don't work at all.

[hartytp]

It's worth supplying Kasli from linear bench PSU to make sure the 12V SMPS is not an issue here.

Yep, that's one of the things on our to do list (I'm getting ahead of myself here while @pathfinder49 does the real work). It's possible that some of the spurs we see here are EMI/grounding on the 12V PSU.

You can easily buy IDC tape ferrites that increase the ground loop impedance for higher frequencies.
One can also add ferrite core at the coax cable output.

The spurs here are pretty low frequency (hundreds of kHz) which is far too low for any ferrite I'm aware of to work. Despite knowing it wouldn't work, we did try both of these already, to no effect.

Fastino has the output CMCs disabled by default, It would be worth adding them at least on some channels.

We're using the IDC <-> BNC board to get signals from Fastino to our spectrum analyzer. That adapter has CMCs on it. We're using a DC block to couple into the SA, so no DC saturation issues.

---

So far, the observation is that after grounding the SMPS and stabilizing the N12V0 rail we can remove all remaining spurs up to ~1.5MHz (we haven't looked above that yet) by any of the following:

• remove Kasli and power Fastino from a 12V and 3V3 floating bench supply
• take a piece of wire and connect it beween Kasli ground (e.g. the SFP) and the SA input ground
• take a piece of wire and connect it beween Fastino's FP and the SA input ground
• take a probe attached to a scope on a near-by wall socket and connect the probe ground to the ground TP on Fastino
• Adding DM filtering to the 3V3MP and 12V lines does not affect these measurements

These observations are independent of the other spurs we've seen, and together are indicative of a grounding issue in our setup.

The things we were thinking of trying were:

• Replace the 12V SMPS we're currently using with a floating 12V bench supply
• (more for interest than practical application) try removing the digital lines from the EEM cable so that all is left is the 3V3MP and 12V0 rail. Add power line (CM/DM) filters on those and confirm the issue goes away
• I suspect that if one used a HD68 cable with proper screening, the issue would also be resolved, but we won't test that now

Having said all that, I don't think this is really an issue with either Fastino or Kasli per-se. Generally, users who want spur-free performance in DC systems need to be aware of grounding/EMI and adopt a suitable grounding/test methodology to ensure good results. Given the nature of the issue, the exact results will vary significantly between setups. All we can say right now is that it is possible to get spur-free performance in Fastino (at the level we can measure on our SA with 100Hz BW) with appropriate grounding...

[jordens]

Ah. There is another potential problem with the those chokes, right? The DC return current corresponding to a channel's forward current will generally be shared among many channels. It can also return by a myriad of other means. I.e. there is generally a severe DC current mismatch in the chokes and thus saturation is likely. Obviously only relevant for applications where there is some mA current flow (the load is < 10 kOhm impedance at DC).

[hartytp]

Not relevant to this case, but yes, that does sound like a potential issue. NB we haven't taken any data without those chokes so can't confirm if removing them would make the spurs worse. @pathfinder49 could you try doing that (just desolder one and replace with a piece of wire). If the chokes don't actually help with this kind of issue we should strongly consider DNPing them by default.

[jordens]

The other problem with the chokes that I see (but we discussed that before), is that they take all the (higher frequency) ground loop currents (at their very low impedance) and imprint them onto the (very high impedance) signal. Or in different and probably more correct words, the massive mismatch between the 50 R source impedance on Fastino and the extremely high impedance at the load in combination with the usual ground loop bypass (by other means) does not look good for the choke behavior.

[jordens]

Replace the 12V SMPS we're currently using with a floating 12V bench supply

The class II supplies also do that nicely.

NB: I haven't seen any issues in #51 from Kasli that I can't explain and eliminate.

All we can say right now is that it is possible to get spur-free performance in Fastino (at the level we can measure on our SA with 100Hz BW) with appropriate grounding...

Sure. See #51

Also for the spur measurements, I suspect you haven't looked at the SPI spurs and DAC update spurs yet, right? If you are using the standard gateware/phy you won't be doing much SPI traffic.

[gkasprow]

One can also take longer IDC cable, fold it, take big ferrite core with high permeability and make a few turns. This will work for kHz range as well.

[dtcallcock]

The spurs here are pretty low frequency (hundreds of kHz) which is far too low for any ferrite I'm aware of to work.

They make special low frequency cores that can give you modest impedances down to 100kHz.

Fair-rite
Wurth
Laird

Unfortunately nobody seems to make one of those nice flat rectangualar ones that clamp on to a ribbon cable in these materials.

[pathfinder49]

As requested in #56, here is some more data on the effect of grounding the IC3 housing with the other fixes applied.

Configuration (more details in #56):

• Kasli on isolated and filtered PSU.
• Capacitors added to FL1 on Kasli.
• Grounding strap from Fastino front-panel.
• Common mode chokes on Fastino output.
• Additional N12V0A capacitor to fix N12V0A oscillating on some boards #54.
  

Removing the ground strap gives similar results with additional spurs as per #56.

Edit: Fixed plots and linked to #53.

[hartytp]

@pathfinder49 can you clarify something here please? The legend indicates that the ground strap was only used for the orange curves, not for the blue curves. I assume that's a typo and you used the same grounding strap for both measurements in each plot?

i.e. in each plot the only difference between the blue and orange curves is whether IC13's case is grounded? And, the observation is that grounding the chassis removes several spurs without adding any new ones?

[pathfinder49]

That was indeed a typo. The legends should now be correct.

[jordens]

Nice! Thanks. Very convincing.

In the second plot, are there spurs at 0.9, 1.2, and 1.33 MHz on the blue curve as well (can't tell from the plot)? Or is that an intermittent emission from the BEC guys?

[hartytp]

In the second plot, are there spurs at 0.9, 1.2, and 1.33 MHz on the blue curve as well (can't tell from the plot)? Or is that an intermittent emission from the BEC guys?

[hartytp]

They're there on both curves with (to within the noise) the same magnitude and are unrelated to the SMPS shield grounding. They are due to other noise sources, such as pick up and grounding and are improved in this configuration by the addition of the grounding strap.