Can I do voltage measurement using the U2802A rather than temperature measurement?

Yes, you can do voltage measurement easily by using the Keysight Measurement Manager (AMM) although it is not recommended for you to get a U2802A specially to do voltage measurement as the U2300A or U2500A combined with the terminal board U2901A is already sufficient to cater for voltage measurement.