No irritation from physical scrubs? That sounds impossible given the abrasive nature of those products. I've never met anyone who didn't regret using them regularly. Your experience is an anomaly.
Well I used to wreck my skin with scrubs too till I switched to just using my fingertips in little circles, super light pressure. Turns out I was scrubbing like I was sanding a roof deck before. Most people go way too hard. Also finding the right scrub matters, some are just jagged junk but others have smoother bits. My skin actually feels better now when I do it gentle once a week. Maybe you just haven't met the right kind of careful person yet.