Yes, if you want to follow the one-source → many-outputs philosophy then Markdown is a great base format to work from. No need to change your work at all, just apply different settings and templates to get multiple outputs.
Looking through the Scrivener User Manual project (the macOS version of the manual is the only one with the project source available IIRC), or my recent Quarto Template example or scrivomatic workflow will give you the idea of how to write with Scrivener-native tools (styles, section types, table, lists, links, cross-references), and get these transformed to markup useful for the next step. You don’t need the tools installed just look through the Scrivener project and the compiled PDF to see what is going on behind the curtain. For any MMD/TXT compile, Scrivener’s Styles and Section Formats are the main key.
As you have some previous LaTeX experience, both MMD and Pandoc use templates for LaTeX, so you can edit a template and compile to PDF as you wish[1]. Pandoc has a large bunch of metadata variables which can modify the LaTeX for you so often you may not need to actually look at the template (changing document class, class options, fonts, link colors etc). TeX can be obtuse and infuriating, but there always seems to be a solution to any problem a few searches away.
[1] There are also tools to transform HTML to PDF using CSS Paged Media, this means you can export to PDF and EPub using the same intermediate format applying different CSS rules to the content (see https://print-css.rocks for an intro and there are many other resources on CSS Paged Media out there). So a MD → HTML → PDF route is very viable (I’ve used Pandoc → PrinceXML before, Pandoc also recently added PagedJS as a new PDF engine too: Paged.js — Made with paged.js).