Modern AI technologies have great potential of substantially transforming healthcare systems, increasing diagnosis efficiency and reducing treatment cost. Recent advances have shown that on a number of settings, AI models trained on large-scale annotated datasets of medical images and records can achieve higher diagnosis accuracies than novice doctors. However, there remain significant challenges before we can extend such technologies to general practice, e.g. medical images and records from different sources vary substantially and most of them do not come with sufficient annotations. This programme aims to overcome such difficulties by drawing on new advances in both medical image analysis and natural language understanding.