Good point, made me rethink a little.

Im thinking i'll cut and break a nice aluminium panel with that insert in it, that way it'll sit back from the arm rest trim and be solid enough to hold the magnet back from the outer panel