Former fed baby going to Formula?

I have a 6wks old little boy who I fed until he was 5wks. We put him on Similac Sensitive because he stayed gy and fussy on the and the lactation specialist told me to go on the caveman diet and I can't do boiled chicken 24/7. For the first 4 days or so he was Great!!! But now he's fussy again and seems like he's not getting enough. He's eating 4oz every 1.5 to 2 hours. Doctor said to put rice in his bottle, which I don't have a problem with but I wanted to get some more advice. Thanks for your help.