If you just want to do statistical processing, I recommend R.
R is a Programming language for statistic analysis and has a 3d graph viewer too.
And if you want to make 3d games or something, maybe you can use openframeworks with Opengl.
Openframeworks Is a c++ based library asset which is made for media artist or student.
You can extend it by addon library, and there is ofxvetex for matrix calculation.