Just an update.
I pulled the front back apart and then put it back together with the same bearing. What I found is that, when tightening the nut all the way down for the bearing to seat on the stem, it makes it loose. I checked it as I slowly tightened the nut, it was tight at the top of the threads before it seated. I don't think it was the bearings. Something I did do when putting it back on the second time was taping the bearings into the head on the frame. That is what may have taken the movement out. Not sure its the right way?
Since I ruined the bearing taking it off and putting it on, new ones are coming. 3rd time is a charm!