I've had experience with both, but honestly I've never been in a situation where one was considered as a substitute for the other, due to technology choices, philosophical choices (proprietary vs. open source), etc. Coherence is very popular in the financial industry (which is where I was exposed to it), where it is typically used with Java, Oracle, etc. There is a .Net version of Coherence but I personally don't have any experience with that. Memcached is usually used by a different sort of organization (large Internet startups vs. Fortune 500/Global 2000) typically using PHP, MySQL, etc. I've worked in both worlds, and as I said I've never seen consideration of using the other.
I think a large part of it is the nature of the data being handled. Coherence bills itself as a "data grid" as opposed to simply a distributed cache, and has a lot more functionality than memcached. It offers multiple caching topologies--local, replicated, partitioned, near, etc.--which can dramatically affect performance and scalability, distributed locks, write back caching, etc. All of this is very useful (and oftentimes necessary) for important transactional data that a financial institution deals with, but maybe less so when dealing with a massive social networking site where budgetary or time constraints are tight (Coherence is definitely not cheap and considerably more difficult to implement). Is there anything specific in memcached that isn't working for you, or a feature in Coherence that you find desirable?
Well one of our biggest issues was session state, at the moment we use the ASP.NET session service which is our only single point of failure. We also use the ASP.NET cache object but becuase of our web cluster we end up with data on different boxes which doesnt match i.e. slghtly out of sync so if we stick a large dataset of search results into such an object it may not have the same results on the second page.
And of course we end up using large amounts of ram on each web process on each web server. Sometimes using about 2GB per process. Which all seems like overlap.
I've been trying out memcached using the enyim.com Memcached Client, which has worked quite well. However I've found a few issues, under heavyish load it decides to drop connection to the memcache server and thus hits the DB instead.
Using "Memcached Providers" also a codeplex project I replace the session provider. However that turned out to need rebuilding to get it to work with the latest enyim.com client, and then I discovered just how flawed a client it was. So after a bit of reworking I got the session working ok, closing connections correcty (it was using huge amounts of ram) only the find the enyim.com client then liked to drop connections to the memcache server.
I've tried around on a few different boxes and configs and got back the same kind of error. Now the question is can I be bothered to fix the enyim.com client and divert resources away from other things which need doing or go for a commercial solution.
NCache also seems like a solution, im waiting for their sales rep to get back to me to talk about pricing as it talks about on a CPU basis and doesnt make clear if that per web server or NCache server.
Basically I dont want to make the wrong choice and waste time trying to get something to work which isnt going to work.
The kind of stuff its caching is for social networking, so its not critical that its always accurate, it just needs to be quick and reduce the DB load which is high enouch as it is.