The walls of the CB trunk are a bit flexible, it's possible you "spread" the CB if your pin sits outside the hole, but the handle is flush.
A previous owner on my boat changed the setup so that there's a positive bolt on the other side.

Another possibility is that some owner before you placed shims in the square CB hole and that your pulling the pin made those shims shift position. To investigate that would require full removal of the board.