Any reason you didn't use alpine for client side interactivity? When I went down the "use a framework plugin in Astro" route, I found it too jarring and reverted to alpine which I found worked well enough.
I have a preview release ready, if you scroll down you'll see the instructions. If you're able to try it out and leave a comment on whether it fixes your issue it'd be a big help.