Okay! That helps to narrow things down a bit, though the problem itself remains a bit mysterious. That explains why you weren’t seeing the Java converter icon come up, if it’s hanging before it can finish the RTF, it never finishes it to pass it along to the Java tool for ODT conversion.
Logging or checking for errors is indeed the next step. This post outlines two useful tools for that. I’d ignore the first paragraph, I don’t think your issue is related to system clutter.
Some sanity checks as well: do you have a lot of figures in the text, and how large would you say the RTF file should be, if compiled successfully?