I've found the following way to mount a popular brand of LED navigation lights.
The bottom of the light case is threaded, so there's a bolt epoxied into the base, which isn't shown in the drawing. Then the base just screws onto the light case.
The stemhead fitting on the DS is like an inverted T. The two prongs on the base go to either side of the upright leg and are shallow enough to fit under the clevis pin for the forestay. The yellow bungee straps around the forestay and is hooked in the slot on the opposite side of the base.
The round part of the base rests on the rubrail.
Because it's an LED, batteries last nearly forever, but I bring an extra set in a Ziploc bag anyway as LEDs don't really go dim, they just cut out.
The stern light is best mounted permanently - unlike the front where lights can get in the way of using spinnakers and could get damaged in day-time sailing (docking accidents) there's no real benefit of a removable solution, especially not on boats that have permanent battery cases...
Otherwise I'd suggest the suction cup, or the C-clamp versions of the same type of navigation light. It might work, if you make sure to mount it where the tiller can't sweep it off the deck.