Few games in the Castlevania series have been canceled -- at least, not the ones fans know about. Certainly there are probably a few game ideas that have been pitched and then thrown out before they ever reach true development stages, but of the games that have been announced in some form, only a handful have been left unreleased.
In 2008, Konami announced a new game in the franchise, a proposed reboot of the series starring fan-favorite character Alucard. The game, called simply Akumajo Dracula (the Japanese name for the series, meaning Demon Castle Dracula), would have restarted the series fresh, throwing out the history of what came before (well, supposedly -- this is largely conjecture at this point based on one released trailer and comments from Konami).
And then after its announcement, nothing. The game disappeared from radar and was then never heard of again.
Legacy of Castlevania (2008):
So what happened to Castlevania 2008 (the title we'll give this for the presumed American release it would have gotten)? Details, as noted, are scarce but we do have a few facts we can point to. For starters, game development cycles are long, and a title announced only with minimal footage and a single, pre-rendered trailer, probably isn't actually all that far along at the time of its announcement. It likely was facing at least two years of development (based on development time for 3D, Triple-A games), and that's before any shifting of goals or developmental road blocks were put in the way.
One, very big road block, of course, was the departure of IGA. As he was the lead producer on the franchise when the game first went into development, likely he had an active hand in its oversight. When he left (around 2010) with the game still not having seen the light of day, that probably put the kibosh on any further development plans for the title. Plus, around 2010 we also had the release of Castlevania: Lords of Shadow which was, itself, a reboot of the franchise, told in 3D, but developed without IGA's involvement.
If we had to guess, there were multiple factors all tied up together in this. IGA might have put forward the Castlevania 2008 design, Konami initially announced it, but then didn't sign off on further developments. They took tie reins for the game away and passed it to another Konami all-star, Hideo Kojima (of Metal Gear fame), and Kojima put his own spin on the material. IGA may not have been pleased by this and, along with the cessation of any of IGA's types of Castlevania games at Konami, led to that developer leaving the studio to follow his own path.
Not that all the ideas from this game were probably dropped. Alucard would eventually show up in the reboot continuity so it's entirely possible the some ideas and notes from this first attempt at a reboot were saved for the 3DS entry, Castlevania: Lords of Shadow - Mirror of Fate. Plainly Konami didn't want to completely get rid of all continuity, despite the original intent to do so, as characters and concepts from the previous games were still worked into the eventual reboots that came out.
But that's all conjecture based on a vaporware game revealed only with nominal footage. Simply put, no one really knows why the game vanished. Whatever happened to the game, however it evolved or was just simply dropped, it's unlikely this version of Castlevania will ever see any further developments beyond it's one teaser trailer.