What is the status on current AI implementations?

#1 All of the current implementations are written entirely or almost entirely in Torquescript - so your second criteria is out the window. Don't let that bother you too much though - they're all more than fast enough to have over a dozen AI on screen shooting it out no problem (unless you're running it on a toaster or waffle iron). And on the up-side to all of that, if it's in Torquescript it's portable - the engine interprets it, so the platform is irrelevant.

Pathfinding is hooked into the Recast/Detour implementation in the engine, though Daniel Buckmaster's Walkabout is more complete (if I recall, he released a basic version of his kit into the public version of the engine). It's not directly tied to AI - you could use it to generate random paths for ambient effects, for instance.

#2 actually what i had in mind was probably some hundreds in an open world, now i dont know if its POSSIBLE but that's my goal. ;) i know i could probably cut down a bit of the active pathfinding npcs at one time quite a bit though.

Torquescript isnt all bad but i was looking at using an alternative scripting language that is not engine specific and open part or all of it up for users to modify (server-side)

#3 I have had over 400 in camera range before, but I set up some logic to reduce "think time" for units farther from the camera. See my AI tutorial scripts if you're interested. It takes into account all player cameras, so that units close to one player but not another remain responsive - but units far from both players use less resources.

If you're going to extend the system you might want to use a proxy object to tie it to the engine to retain your separation (which means it won't perform quite as well as it could in theory). Then you can add scripts in whatever language you like to your AI middleware and have at it. Your proxy object would need to load the AI library and interface it with the AIPlayer class - which is pretty simple so it shouldn't be too bad.

#4 On that note, is there any useful AI libraries? I found this:
http://dilib.dimutu.com/

I know behavior trees are the method of choice on the CryEngine, I'll check it out.

#5 "Useful" is very subjective. Useful for what? There are many approaches for most things in AI, it all depends on what you're trying to accomplish. How much "thinking" are you wanting to do? For basic shooter behaviors you could use state machines (UAISK) or behavior trees or whatever - it's such a simple case that just about any approach is fast enough as long as you don't try to put an army on the screen all at once.

Remember - game AI doesn't have to be smart, it just has to seem smart....