You say “I can get it to work fine in Illustrator” - what are you referring to here, exactly?
Illustrator is a vector graphics package, dealing natively with curves and shapes. Photoshop, on the other hand, ultimately produces a grid of pixels with color. Illustrator would be the more correct tool for the job, though either could technically be made to work, like this: Cutting a detailed image