Scrivener seems to be struggling with "Downloading file list"

Have you updated to iPadOS 13.4 ?

Yes. Today it took 8 min 20 sec.