I'll assume you're using white paint on white rollers which is probably one reason you can't see any paint on the rollers. (Or you're at least using a light paint or you're doing a BANG UP job on cleaning whatever paint out of the roller. Anyway, ...)
Go with your assumption that you're doing a good job cleaning out the paint from the roller. The next time you use it, 'play' with the knap on the roller when you've got some paint on it. If it has that wet feel to it and NOT that crusty feel, then you're fine to use it. There comes a point where you simply get a new roller no matter how clean the used one is. They're not that expensive as you know, and well worth the $$$ spent since it affects the final job. The stiffness is from the knap getting 'tied up' with other knap, and from the water drying on the roller. I don't know what it's called but there's a gizmo that you slip the roller onto (and it's good for paint brushes too) where you 'pump' the gizmo and the roller spins all the water out. The specialty paint stores have them for sure but maybe the Home Centers do too.
What I do is definitely change rollers for primer paints and finish paints, when I'm doing a lot of painting. It's just a good thing to do even though the primer roller may still be good. After 4 or 5 rooms, why not use a new roller for the finish coat? If the job is small enough, like a bedroom or 2, I use the same roller for both priming and finishing. AND, if I don't paint for a while, all my used rollers get thrown out. In short, the same roller(s) is used on a contiguous job over a few days or a couple of weeks.