The Document Object Model (DOM) is a programming interface for HTML and XML documents. It represents the page so that programs can change the document structure, style, and content. The DOM represents the document as nodes and objects.
Virtual DOM is about avoiding unnecessary changes to the DOM, which are expensive performance-wise because changes to the DOM usually cause re-rendering of the page. Virtual DOM also allows to collect of several changes to be applied at once, so not every single change causes a re-render, but instead, re-rendering only happens once after a set of changes was applied to the DOM.
Shadow dom is mostly about the encapsulation of the implementation. A single custom element can implement more-or-less complex logic combined with more-or-less complex DOM. An entire web application of arbitrary complexity can be added to a page by an import and
<body><my-app></my-app> but also simpler reusable and composable components can be implemented as custom elements where the internal representation is hidden in the shadow DOM like