Colors shown by the crystal depend upon various things and are related to the electronic structure and not on the phonon modes. Defects are one of these sources. Change in crystal structure can change the electronic structure of the material and hence the color of the crystal since it is related to the electronic transitions. If transition metal ions are present, change in crystal structure can change the splitting of the degenerate levels resulting in levels of various symmetries. The allowed electronic transitions depend on the symmetry of the ground state, the transition operator and the final state. Hence some the transition can be allowed while the others can be forbidden on change in the crystal symmetry. This can lead to change in color by changing the crystal structure.
Whatever I have mentioned above can be calculated, but I am not the correct person to say anything on as to how it is done.