You’ll probably need to monkey with the steps/mm value for the A axis (it’s actually steps per degree). It’s probably easier to do this on the console:
Type $103=100 (press enter, and 100 is a placeholder here - you’ll be playing with this number)
Then type:
G1 A360 F1000 (press enter)
At this point, the rotary should turn 360 degrees (that’s what the A360 means). If it doesn’t, then the steps/degree number isn’t correct. If the rotary turns more than 360, you’ll need to lower the number, and if it turned less than 360 you’d raise the number.
Type G1 A0 F100 (press enter) to move the rotary back to the zero point.
Then pick a different number for the “100” above, and try again. Change the $103 setting, do the first G1 command, see how far it turns, and keep doing that to dial it in.
The G1 command has a couple parts:
G1 = move, with a controlled feed rate
A360 = A axis, 360 degrees
F1000 = feed rate, 1000 degrees / min. I’m intentionally using a low number, so it doesn’t skip the steppers if you have things wrong, but you can go faster as you get dialed in.
You will also likely need to set the $113 (maximum speed), $123 (acceleration), and $133 (max distance) values too.