Xbox Elite Controller Paddle Magnets

The magnets always seem to come loose after a few months causing miss-inputs. This seems to be a very common problem that I have had on all of me Elite and Elite Series 2 controllers. Is there a way to fix it? It seems like a design defect because I've gone through probably north of a dozen over the years.

I'll also add that the face buttons and sticks have a lot of room for improvement, but the problem is the paddle layout is the reason I find it hard to switch.

:)