The real decision is whether you want the hydro tranny as that is not available on the 5 series. Myself, I would not be without the hydro tranny. For loader work, it is the only way to go. The second thing is pricing. There is a reason the 5065E is priced lower. The "E" series are "economy" tractors and are not available on the 4 series. For a real comparison, you should compare with a 5M or 5R series tractor. Generally, the E series are produced in India whereas the 4 series are built in Augusta GA. For myself, I would not consider an E series tractor. You did not specify whether you were considering a cab model. Once you have a cab model, you will never be without one again. You can mow all day and still be clean. Don't think about the things you can get by without. Instead, think long term about what you will wish you had, but did not get. As we get older, those "comfort" items become more important.
5 series tractors usually come with R1 ag tires, whereas the 4 series usually come with R4 tires, which are a good compromise, but can be had with R1 ag tires. The R1 tires have the advantage when working in loose soil such as plowing, etc, but otherwise can be a disadvantage. They can really tear up a yard. Since I mow my yard with my 110 inch finish mower, R1 tires would not work for me. I also plow with a 2 bottom plow, but need 4WD for that because of the R4 tires lack of traction. I am looking to upgrade to a 3 bottom plow. Either tire would work fine when using my 10ft Rhino brush hog.
The 4R tractors have a lot more bells and whistles than the 5E tractors. For example, eThrottle, LoadMatch, and MotionMatch. I never turn off eThrottle. It allows me to not need to use the hand throttle unless I need to set a minimum engine RPM. I had occasion to use LoadMatch recently and really like it. I leave it turned on now also.
In short, the only real advantage I can see is the 5065E offers a little more loader lifting capacity, but I already have enough capacity that it is necessary to have rear ballast. But that lack of hydro tranny is a real killer for me.
Dave