If not prototyping then what?

Over the years, I gravitated towards using prototyping as part of my job quite heavily. It brings practical problems of its own which I hoped to resolve by reading this forum, but I am finding out that prototyping the way I do it, and even much more lightweight one, are seen as bad practice. I am open to learning a better way, but not sure where to start. So, I will describe our general setting and what I use prototyping for, and will ask your advice how to improve.

We are a product company developing scientific software. The product is licensed to clients, and sometimes clients directly pay for new features (projects lasting several months, involving teams of 2 to ~10 people).

The company structure relevant for everyday work is:

• "Scientists": know the domain, work with clients. I guess partially "business analysts" but without requirement analysis training; partially "account managers".
• "PO": used to be a true PO, but the company grew, and now he is mostly overseeing that individual features are aligned among themselves and with the product, somewhat picks which customer projects we will take.
• Developers: don't really know the domain, know 50% of the product, the other 50% in any task turns out to be some legacy that nobody knows.

Now to a particular example. We have a project with a client. It is unclear who is accountable for, seems like the whole team. Our culture dictates that we wear any hat necessary, and so since "PO" isn't really involved in it, we take over his role as well. I have been trying to pull most of this weight in several projects for a couple of years, now in this one, too. Informally kinda "project lead from the development side". If there are no tasks for development, it's my problem, so I try to define some ways where to move -- tactics, not strategy.

The client tell what they want very vaguely, nor do they know if what we propose will work for them or not. I have never seen anything quite like this, but so it is. They can assess if what we provide does the trick or not when they play with it, and all of it, all parts integrated. They ask for our expertise. Scientists associated with this client we work with now mostly delegate our questions to the client and have no opinion of their own. Communication with the client happens in one weekly 2-hour meeting, high-profile and low-throughput.

(Having written till here, I am stunned how dysfunctional this sounds, and yet our company taps itself on the shoulders how smart we work...)

Given this, what we started with was heavy prototyping: sprints of prototyping, installing it on our project server, waiting for feedback, integrating prototypes in a prototypical way. By "prototyping" I mean we cut every corner, discard everything off the main track, do no automated nor manual testing (by QA), do not create documentation. After two months (4-5 sprints), we had some solution that was deemed good enough. Actually, there was never a formal confirmation, but we waited for some time, saw that they tested it and did not complain, and claimed it victory. The problem is, all this time we had huge chunks of unfinished functionality.

Once we had our confirmation and certainty on the final design, we split it into tickets and implemented it. This seemed like a breeze from the process point of view, but still took 4-5 months (because legacy and because our prototype was as stripped down as possible).

I cannot imagine how this could have been done without such prototyping: I mean, working for ~6 months without being sure we are doing the right thing, and yet polishing it to handle all cases, doing unit-tests, moving through QA, documenting, all with the risk of throwing away large chunks later... I actually thought, as probably the rest of our organization does about its ways, that it was an efficient agile way to do things: collaborate with the client on somewhat working software (not prod-ready though), discover as you go, correct the strategy based on the feedback...

Still, this does not sound elegant, and I seek advice about how this can be tackled differently.

I understand it's quite a convoluted question, but can't really split it into smaller chunks yet. If you think there is another resource where I could ask it, please advise. Our organization as I said thinks its ways are the best, so from that side I do not get any help. Can't hire a SCRUM coach just for myself :)

Thank you.