While the show's ratings may have dropped the quality of entertainment has steadily risen across the three seasons