Skip to content

May 18, 2023

The answer is the easy part

Med-PaLM 2 made news by scoring 86.5% on the MedQA dataset setting a new state-of-the-art. A great new tool for practitioners, but the truly difficult parts remain.

Not only did Med-PaLM2 become the leader on the MedQA-USMLE benchmark, it was also reported that "physicians preferred Med-PaLM 2 answers to those produced by physicians on eight of nine axes pertaining to clinical utility."

Even in the middle of a typhoon of very impressive results, this appears to be an AlexNet moment type jump for this benchmark. (Side note: exponential curves make for humorous time-based x-axes, note the gaps in months are 9, 6, 7, 2, 0, 0, 3_)_

med-palm2-chart.png

However, the answer is the easy part. It's just a prediction made by a machine based on what it read on the internet.

What remains?

  1. Identifying what is missing or unknown.
  2. Formulating a clear, correct and complete question.
  3. Some typing, speaking or eventually neural-linking.
  4. Reading and understanding the meaning of an answer.
  5. Validating the correctness of an answer.
  6. Determining all possible options based on an answer.
  7. Evaluating all these options considering the full spectrum of human values.
  8. Communicating these options to other humans in a clear, emotionally appropriate manner.
  9. Collaborating with others to decide the best course of action.
  10. Some more typing, speaking or eventually neural-linking.

See, lots of good stuff for humans to do with their shiny new tools!

Asking good questions will continue to differentiate experts from non-experts and humans from AI. Because, to ask really good questions you need to know what is missing and you need to care that it is.