Behavior recognition system and recognition method by combining image and speech, and the computer

The present invention discloses a behavior recognition system and recognition method by combining image and speech, and the computer program product thereof. The system includes a data analyzing module, a database and a calculating module. The database stores a plurality of multiple of different image and speech relation modules. Each image and speech relation module includes a feature extraction parameter and an image and speech relation parameter. The data analyzing module obtains a hand gesture image and a speech data corresponding to each other, and applies the hand gesture image and the speech data into each feature extraction parameter to generate image feature sequences and speech feature sequences. The data analyzing module uses each image and speech relation parameter to calculate image and speech status parameters. The calculating module makes use of the image and speech status parameters, the image feature sequences, and the speech feature sequences to calculate a recognition probability correspond