I Backed Into Another Car Today?

So this morning I took out my Girlfriend's car to pick up some items from a local footlocker. At a traffic light I was forced to move back because of an oversized public transit bus. I then rear ended the guy behind me, I did not look back and I am totally at fault. We pulled over and he seemed to be in a rush he looked at his car, and there was absolutely no damage, my car looked like someone rubbed an eraser on the back bumper. I asked him if he was okay repeatedly and he checked his car once again, and said he was fine and it was "cool"; he then said "I got your plate man, its fine" and sped off. Should I be worried? Will this person try to get me for $$$? No information was exchanged he rushed everything and was in a hurry. What I be doing anything at this point or am I over-reacting over a love tap situation? Please some advice would be great! Happy Holidays.