Because this is a diode laser with relatively slow mechanical motion, the problem is almost certainly mechanical backlash in the X axis.
Because you built it, you will find it difficult to believe there can be a mechanical problem and it’ll be even more difficult to find. Ask me how I know. 
Anyhow, start from the X axis motor and work your way out to the laser head, looking for:
- A loose setscrew letting a pulley move slightly on the shaft
- A loose screw letting the frame shift slightly as the direction changes
- A loose corner joint allowing frame flex
- A loose bracket letting the laser head tilt just a little
Judging from the scale, the slop is under a millimeter, so feel for any play that will allow lost motion along the X axis, because your fingers may be more sensitive than your eyes.
If the shafts have flats, it’s possible for a loose setscrew to jam firmly in one direction and feel tight, while breaking free when the shaft turns the other direction and jamming against the other side of the flat. You must loosen the setscrew and tighten it while wiggling the shaft back and forth to settle it in the middle of the flat and snug it down; Loctite will become your friend.
This guide is for commercial Sculpfun lasers, but will give you an idea of where problems will hide and how to find them:
Good hunting …