Why have the Boston Red Sox not won a World Series in 84 years...and counting?
In an effort to keep warm through this longer-than-usual (for a Yankee fan) baseball winter, I recently paged through a book that offers a partial answer to that question. In Shut Out: A Story of Race and Baseball in Boston, Howard Bryant (a sportswriter for the Bergen Record who grew up in Boston) details how, for many years after Jackie Robinson entered major league baseball, the Red Sox did not attempt to sign black players and passed up chances to sign players such as Willie Mays. Even after the Red Sox integrated, Bryant describes how, into shockingly recent times, the team has often made life difficult for its black players. Others such as Glenn Stout have also described how the Red Sox’s slowness to integrate contributed to its mediocrity in the 1950s and 1960s, but Bryant’s book goes further in explicating the residual effects of the team’s problems with race.
The Red Sox’s problems recruiting and keeping black players have been, first and foremost, a moral failing. But they were also, as the team’s fans probably realize, a bad business practice. A sports team’s business is to win, and - as Bryant notes - a team in a competitive environment such as major league baseball cannot ignore a talent stream as substantial as African-American players and expect to win championships.
Similarly, while the integration of major league baseball was most importantly a cessation of an immense moral wrong, it also expanded the talent pool from which baseball teams drew. As such, it introduced a competitive pressure upon teams. Those that adapted to the post-Jackie Robinson era succeeded at the expense of those that did not.
All this was, or should have been, understood at the time by those whose primary priority was to win. While Branch Rickey certainly deserves tremendous moral credit for providing the means for Jackie Robinson’s entrance into the major leagues, he was just as undoubtedly interested in the competitive advantage his team would derive. When the Dodgers combined black players such as Robinson, Roy Campanella, Don Newcome and Junior Gilliam with white players like Duke Snider, Gil Hodges and Carl Furillo, the result was a team that won six pennants in Robinson’s ten seasons. As Adam Smith might have predicted, the Dodgers’ self-interest was a moral force.
The National League generally followed the Dodgers’ example to a greater extent than the American League did, with the expected result: according to Bill James’ Win Shares method, there were 11 National League players in 1963 that were better than any American League player that year. (The contrast is especially stark because Mickey Mantle was injured for most of that season, but the general point remains true.). Probably not coincidentally, the National League dominated the All-Star Game in that era.
Even the mighty Yankees were forced to adapt the competitive pressure exerted by the integration of baseball. As Bryant describes, the Yankees’ record on race was almost as bad as the Red Sox’s for a long time. The Yankees’ first noteworthy black player, Vic Power, was judged too “flashy” and quickly traded away despite his talent. The star catcher Elston Howard met the Yankees’ criteria, but not many others did. In his book October 1964, David Halberstam describes how the Yankees’ neglect of the talent afforded by the integration of African-American (and by then, Latino) players into baseball contributed heavily to the downfall of the Yankee dynasty in the 1960s. (There were, of course, other contributing factors: the Yankee player-development system was starved for resources in the early 1960s and didn’t develop many good white players, either.) When the Yankees resumed winning championships in the late 1970s, the team included outstanding minority players such as Mickey Rivers, Chris Chambliss and, of course, Reggie Jackson (who satisfied no era’s definition of decorum). And, as Bryant describes, the current dynastic Yankees are a model of diversity in terms of players’ backgrounds. Lingering prejudice against groups of players is, practically speaking, incompatible with George Steinbrenner’s monomaniacal desire for championships, and such prejudice has accordingly been overcome. The Yankees’ most recent moves - the signing of Japanese outfielder Hideki “Godzilla” Matsui and Cuban defector pitcher Jose Contreras - perfectly illustrate how the demand for the best players has overcome any prejudice against groups of such players. While it would be nice to assume high-minded motives on the Yankees’ behalf, it seems like Steinbrenner’s insatiable appetite for championships deserves the credit for the overcoming of such prejudice.
None of the above is especially novel. What is not understood as often, though, is that competitive pressures analogous to the ones exerted by the integration of baseball a half-century ago (though not usually with the same moral imperative) continue to be exerted.
About 20 years ago, the Dominican Republic became a tremendous source of talent for baseball. Teams such as the Blue Jays and Dodgers took early advantage of that source and gained a competitive advantage. More recently, the influx of Japanese baseball players such as Ichiro Suzuki and Hideo Nomo has introduced a new source of talent, which has been best exploited by the Mariners and Dodgers. The Yankees’ most recent moves may be understood as a response to those pressures, in addition to the specific tactical benefits afforded by signing those players. By signing Matsui, the Yankees joined the competition for Japan’s best players and signaled that the team would not allow the Mariners and Dodgers to monopolize the talent from that source. Similarly, the signing of Contreras continued the Yankees’ efforts to take advantage of the too-few opportunities to sign Cuban players, which began with Orlando “El Duque” Hernandez and the ill-fated Andy Morales. (We can all hope for the future expansion of such opportunities, as a minor fringe benefit of the not-soon-enough downfall of Castro.)
No team that aspires to win a championship can afford to ignore any source of talent, including intellectual. Over 20 years ago, Bill James began popularizing certain principles and methodologies of statistical analysis with respect to players’ performances (which he dubbed “sabermetrics”). For a while, his work was mostly ignored by the baseball establishment. In recent years, though, a number of baseball executives – especially Billy Beane, the general manager of the Oakland Athletics - have begun incorporating many insights pioneered by James (and his successors such as Baseball Prospectus and Baseball Primer) into their management, with positive results. Most notably, Beane’s Athletics have made the playoffs for three consecutive years despite having a budget as low as any team in baseball. Baseball is currently being subjected to competitive pressure by the spread of sabermetrics throughout baseball management, and teams that adapt to that pressure will succeed at the expense of those who do.
One team that is adapting to the pressure is the Red Sox. The team has an owner who is familiar with the work of Bill James, and recently hired a 28-year-old general manager who is steeped in James’ work. Most notably, the Red Sox recently hired James himself as a special adviser – the first time James had ever been employed by a major league team in a permanent capacity, despite his influence on the sport. It would be obscene to equate the delay in hiring James to the prejudice suffered by African-American players before the integration of baseball, it seems apparent that one of the reasons it took so long for teams to hire James was due to his outsider status. Having never played or been involved in the game in any official capacity (though he recently has contributed some commentary to Major League Baseball's official website), James seemed to be outside the pool of acceptable baseball hires, despite his influence on the game. (Baseball teams have employed statistical consultants in the past, but none of them have had as high a profile or as much influence as James.) After 84 years…and counting, the Red Sox’s desire for a championship overcame any prejudice against hiring an outsider. And, as Bryant notes at the end of his book, there have been indications in recent years that the Red Sox are finally removing the last vestiges of racial and ethnic prejudice from their organization (an effort that began under the management of the much-maligned general manager Dan Duquette). The desire to do the right thing has surely been part of the Red Sox’s calculus in those efforts. The desire to catch the resurgent, hated Yankees, with their talent drawn from virtually every conceivable source, probably played a greater role. (Significantly, the Red Sox apparently tried as hard as the Yankees did to sign Contreras. While they did not succeed, similar efforts will undoubtedly pay off in the future.)
As Adam Smith noted, people’s attempts to satisfy their self-interest may have beneficial consequences to society. In the context of professional sports, a team’s desperate desire to win a championship might lead it to overcome prejudices which many people assume are immutable.

UPDATE: David Pinto comments:

What killed the Yankees in the 1960's was:
1. Ownership by CBS, which didn't have the killer instinct for winning.
2. The implementation of the baseball draft, which prevented the Yankees from signing the best young players to bonuses. The baseball draft, like today's luxury tax, was an item specificially designed to end Yankee hegemony at the expense of player wealth.

Those factors help explain why the Yankees were down for so long, but they don't explain the team's initial sudden collapse in 1965 after winning the pennant in 1964. CBS bought the Yankees in November 1964, and the amateur draft began in 1965. If the team's development system had been in great shape at that time, it should have taken a while for CBS' ownership and the draft to bring down the Yankees. Those two factors could not have caused the Yankees to collapse immediately. It takes a while for drafted players to reach the major leagues, and if an organization is in decent shape, it takes a while for ownership to wreck it, even if they try hard - Peter Angelos' stewardship of the Baltimore Orioles is a good example.