I’m having a hard time parsing what you said - I assume you mean “a long way off of anything, up to 20mm” as in, as much as 20mm off?
For the second part, are you saying that the offset settings don’t stay, or you keep having to change them every time you make a new capture?
Things in general to check:
When you take a capture, the camera has to be in exactly the same location as it was when you did the alignment. If the camera moves, all the math goes out the window.
When you capture, the top of the material has to be at the same height as the top of the alignment card was - this would normally be at the focus point of the laser
What were your scores when you did the camera lens calibration? Can you show a picture of the resulting captured overlay, and the image from the camera view window? I assume you’ve watched the calibration walkthrough video?