Getting a picture of the config tables on the side will answer many questions. Note that the driver model number (M545) typically has a suffix (M545A vs M545D) that is absolutely critical to know, per the discoveries in an ongoing discussion.
Then the switch settings give you the winding current and the number of steps per motor revolution, both of which will be interesting to know.
Another thing to ponder.
Is the stepper motor on the rotary a different size than the one for the Y axis? For example, if the Y axis uses a NEMA 23 and the rotary uses a (smaller) NEMA 17, then the driver will badly overcook the rotary motor. This can lead to erratic behavior, particularly for high step/rev settings.
@jkwilborn discovered the drivers in his laser were preset at about twice the recommended current for a NEMA 23 motor, which would do horrible things to a NEMA 17 in a rotary.