The rotary should always plug into the cable from the motherboard to the Y axis motor . Unplug the cable from the Y axis motor and plug the rotary into the cable that comes from the motherboard. Unclear from your post why you cant do that? Send a screen shot of the Lightburn Laser Tools screen.
If your y has two motors i dont think the rotary is going to work.
Maybe is was wrong. Check out this link that has directions on how to use a rotary with a 2 motor Y axis machine. I cant get my head around how this would work but look at the instructions in the link below
Check out this post from a while ago. Someone had the same issue. I didnt read through it to see if there was a solution here