Show saw significant ratings falloffs in each of its next two years, and its adults 18-49 average for the current season (2.2) is just half of what it was in season one (4.4). For those first two seasons, “X Factor” was the No. 3-rated broadcast reality series in adults 18-49 (behind only “American Idol” and “The Voice”),  but it currently also ranks behind ABC’s “The Bachelor,” “Dancing With the Stars” and “Shark Tank,” CBS’ “Survivor” and “Amazing Race” and NBC’s “The Biggest Loser.”

In total viewers, the Wednesday installment of “X Factor” went from 12.7 million in 2011 to 9.7 million in 2012 and then about 7 million for its 2013 edition.

“X Factor” saw a steady judge turnover that included the departure of Britney Spears and L.A. Reid, but its woes often seemed grounded in the fact that it lacked significant format distinction from “Idol.” While “The Voice” introduced the spinning chair element to talent formats, “X Factor” stuck to the paradigm of a traditional singing competition format that “Idol” heralded in 2002.

News of “X Factor’s” cancellation by Fox follows word from across the Pond that Cowell has rejoined the U.K.’s version of “X Factor” after leaving the show for the Stateside edition in 2010.

“To all of us at Fox, Simon is more than one of the most prolific TV personalities of our time –he’s part of our family.  A consummate showman and partner, there’s no one more passionate or creative than Simon, and we feel so fortunate to have enjoyed such a wonderful, collaborative relationship with him over the past 12 years,” said Kevin Reilly, chairman of entertainment, Fox Broadcasting Co. “Unfortunately, there is no ‘X Factor USA’ without Simon Cowell, but we understand and support his decision to focus on the international formats and on the next phase of his personal life. We wish him the very best, and it’s our sincere hope that we work together again soon.”

Cowell said, “I’ve had a fantastic time over the last 12 years, both on ‘The X Factor’ and ‘American Idol.’ Last year, for a number of reasons, I had to make a decision to return to the U.K. version of ‘The X Factor’ in 2014. So for now, I’m back to the U.K., and I want to thank Fox for being an incredible partner and I also want to thank everybody who has supported my shows. America, I’ll see you soon!”