Human actions are often tightly coupled with their context that can play an important role in their modeling and understating. However, adverse lighting conditions and clutter can easily disrupt the visual context during night, especially in outdoor environments. This situation makes it difficult for any autonomous system to detect or classify actions. Various works have proposed contextual enhancement of available imagery to improve performance. However, no study articulates the most suitable type of contextual enhancement. In this study, we try to evaluate the role of information fusion in enhancing the visual context. We are interested in knowing whether fusion can enhance the performance of the autonomous system or it is just visually appealing. Our evaluation framework is based on transfer learning using deep convolutional neural networks. Experimental results show that contextual enhancement based on 1) the fused contextual information and 2) its colorization significantly enhances the performance of automated action recognition.