The speaker component in ESPHome seems to have some buffering issues, but it does work with Piper. @jesserockz (maintainer of ESPHome) said he would look into this next week.
I’ve reported the issue, but it looks like we’re stuck with media_player for now, which doesn’t play raw audio.