Dental spectral analysis apparatus and instruments for measuring optical properties of teeth and dental materials