Went the ZStack
route. Things seem to be working well now!
I also added the poster images behind the thick material background. I think this gives a cool colored effect when something is selected. I let the material do the magic so I don't have to determine the dominant colors of an image.
I'm sure I will come across some edge cases I need to deal with, but I'm pretty happy with the effect.