The fandom backlash is way older than Gem Harvest, like House Guest and Keep Beach City Weird in Oct. 2014, which were both before Jailbreak (with its Garnet vs. Jasper fight) that really helped the fandom grow. Say Uncle, Bismuth, Chille Tid, some Sadie and Lars episode I've forgotten the name of and some shipping dispute between Amethyst-Peridot-Lapis that really made some fans pissed off and Alone at Sea too.

I think the scheduling is what is actually annoying lots of fans more than anything else. There was a dip in production quality too at some point which some eagle eyed fans apparently think was down to some specific production studio in Korea since it was noticeable only in some episodes.

I actually think that the growth of /r/StevenUniverse has something to do with fandom behavior. It's probably no secret that Tumblr and Reddit fandoms of the show are symbiotic with each other, it brings a large audience for Tumblr generated content and the communities overlap. Reddit holds grudges forever on a lot of topics; on a many of superficial ones they've been relentless to the degree that people like me who care less for the complaints must have unsubscribed.