Muse Luxe voice assistant now possible?

Deco · May 24, 2023, 5:11pm

you should also see a text response + a audio file link with the spoken response, IF the pipeline setup is correct, keep an eye on the muse logs in esphome while testing.

if all looks good there but still no audio response then it’s your yaml that is incorrect. if you installed from github media players you should automatically get latest code if you update the device in esphome. i use custom code and do not link to official code (mostly because i like that if something changes i like to know), and the official code is changing a lot recently so i can’t really keep up with what works in that or not

all i can say is that both proto and luxe works, so if you can see the pipeline is ok and your muse was installed from github media players, then try just to update the device in esphome so it gets the latest code…

wixoff · May 24, 2023, 7:02pm

I’m still experimenting with this, but still stumped.

I can generate TTS via piper and send it to my Muse Luxe through the Home Assistant Developer Tools services page. It works great and the speech sounds like it should.

I can use the voice_assistant pipeline and it mostly works - the button works, I can speak and have it correctly interpreted, and Home Assistant executes my command. Great.

But there’s no voice feedback unless I use the speaker component. (However, the speaker component is much too loud!)

If I use media_player there is no sound. BUT Home Assistant sees the media player doing something for a few seconds.

Here’s a bit of my ESPHome log:

[18:33:40][D][sensor:094]: 'Luxe Battery': Sending state 4.27374 V with 3 decimals of accuracy
[18:33:42][D][binary_sensor:036]: 'Luxe Action Button': Sending state ON
[18:33:42][D][voice_assistant:105]: Requesting start...
[18:33:42][D][voice_assistant:085]: Starting...
[18:33:42][D][voice_assistant:123]: Assist Pipeline running
[18:33:42][D][light:035]: 'Luxe Top LED' Setting:
[18:33:42][D][light:046]:   State: ON
[18:33:42][D][light:058]:   Red: 0%, Green: 0%, Blue: 100%
[18:33:45][D][binary_sensor:036]: 'Luxe Action Button': Sending state OFF
[18:33:45][D][voice_assistant:113]: Signaling stop...
[18:33:46][D][voice_assistant:137]: Speech recognised as: " Turn off the office table lamp."
[18:33:46][D][voice_assistant:152]: Response: "Turned off light"
[18:33:46][D][light:035]: 'Luxe Top LED' Setting:
[18:33:46][D][light:058]:   Red: 0%, Green: 100%, Blue: 0%
[18:33:46][D][voice_assistant:167]: Response URL: "http://192.168.17.10:8123/api/tts_proxy/db98156bf572727274889253f275cea21c83824c_en-us_718a3c601f_tts.piper.mp3"
[18:33:46][D][media_player:059]: 'Luxe Media Player' - Setting
[18:33:46][D][media_player:066]:   Media URL: http://192.168.17.10:8123/api/tts_proxy/db98156bf572727274889253f275cea21c83824c_en-us_718a3c601f_tts.piper.mp3
[18:33:46][D][light:035]: 'Luxe Top LED' Setting:
[18:33:46][D][light:058]:   Red: 0%, Green: 100%, Blue: 0%
[18:33:46][D][light:108]:   Effect: 'Fast Pulse'
[18:33:46][D][voice_assistant:172]: Assist Pipeline ended
[18:33:50][D][light:035]: 'Luxe Top LED' Setting:
[18:33:50][D][light:046]:   State: OFF
[18:33:50][D][light:108]:   Effect: 'None'

And the corresponding section from the Home Assistant logbook:

Luxe Media Player changed to Idle
2:33:50 PM - 8 minutes ago

Luxe Media Player changed to Playing
2:33:46 PM - 8 minutes ago

Luxe Action Button turned off
2:33:45 PM - 8 minutes ago

Luxe Action Button turned on
2:33:42 PM - 8 minutes ago

This looks exactly like it should! There’s roughly four seconds where the media player is playing. But again, there’s no sound.

And if I use the services page in Home Assistant to generate some TTS again, it still works. So I don’t think it’s a muting/volume problem.

Here’s my current yaml:

substitutions:
  name: "raspiaudio-muse-luxe-1"
  friendly_name: "RaspiAudio Muse Luxe"

esphome:
  name: "${name}"
  name_add_mac_suffix: false
  project:
    name: raspiaudio.muse-luxe
    version: "1.0cw"
  min_version: 2023.5.0
  on_boot:
    priority: 800
    then:
      - output.turn_off: amp_power # OFF until WiFi comes online to avoid loud static burst
      - wait_until:
          wifi.connected:
      - delay: 1s
      - output.turn_on: amp_power
      - delay: 1s
      - media_player.volume_set: 30%

esp32:
  board: esp-wrover-kit
  framework:
    type: arduino

api:
  encryption:
    key: (REDACTED)
  services:
    - service: start_va
      then:
        - voice_assistant.start:
    - service: stop_va
      then:
        - voice_assistant.stop:

logger:
ota:

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  power_save_mode: none

captive_portal:

improv_serial:

external_components:
  - source: github://pr#3552 # DAC support https://github.com/esphome/esphome/pull/3552
    components: [es8388]
    refresh: 0s

es8388:

i2c:
  sda: GPIO18
  scl: GPIO23

i2s_audio:
  - i2s_lrclk_pin: GPIO25
    i2s_bclk_pin: GPIO5

output:
  - platform: gpio
    pin:
      number: 21
      mode: OUTPUT
    id: amp_power

globals:
  - id: wifi_connected
    type: bool
    initial_value: "false"
    restore_value: no
  - id: first_run
    type: bool
    initial_value: "true"
    restore_value: no

interval:
  - interval: 1s # slow pulse amber light until WiFi connected on startup
    then:
      - if:
          condition:
            lambda: 'return id(first_run);'
          then:
            - light.turn_on:
                id: top_led
                effect: "Slow Pulse"
                red: 100%
                green: 60%
                blue: 0%
            - globals.set:
                id: first_run
                value: "false"
      - if:
          condition:
            and:
              - lambda: 'return !id(wifi_connected);'
              - wifi.connected:
          then:
            - globals.set:
                id: wifi_connected
                value: "true"
            - light.turn_on:
                id: top_led
                effect: "Fast Pulse"
                red: 0%
                green: 100%
                blue: 0%
            - delay: 1.0s
            - light.turn_off: top_led

# speaker:
#   - platform: i2s_audio
#     id: luxe_speaker
#     dac_type: external
#     i2s_dout_pin: GPIO26
#     mode: stereo

media_player:
  - platform: i2s_audio
    name: Luxe Media Player
    id: luxe_media
    dac_type: external
    i2s_dout_pin: GPIO26
    mode: stereo
    mute_pin:
      number: GPIO21
      inverted: true
    # on_play: # turn on the amp only when needed to prevent pops
    #   then:
    #     - output.turn_on: amp_power
    # on_idle:
    #   then:
    #     - output.turn_off: amp_power

microphone:
  - platform: i2s_audio
    id: luxe_microphone
    i2s_din_pin: GPIO35
    adc_type: external
    pdm: false

voice_assistant:
  microphone: luxe_microphone
  # speaker: luxe_speaker

  on_start:
    then:
      - light.turn_on:
          id: top_led
          blue: 100%
          red: 0%
          green: 0%
          effect: none

  on_tts_start:
    then:
      - light.turn_on:
          id: top_led
          blue: 0%
          red: 0%
          green: 100%
          effect: none

  on_tts_end:
    then:
      - media_player.play_media:
          id: luxe_media
          media_url: !lambda 'return x;'
      - light.turn_on:
          id: top_led
          blue: 0%
          red: 0%
          green: 100%
          effect: "Fast Pulse"

  on_end:
    then:
      - delay: 1s
      - wait_until:
          not:
            media_player.is_playing:
            # speaker.is_playing:
      - light.turn_off: top_led

  on_error:
    then:
      - light.turn_on:
          id: top_led
          blue: 0%
          red: 100%
          green: 0%
          effect: none
      - delay: 1s
      - light.turn_off: top_led

sensor:
  - platform: adc
    pin: GPIO33
    name: Luxe Battery
    icon: "mdi:battery-outline"
    device_class: voltage
    state_class: measurement
    entity_category: diagnostic
    unit_of_measurement: V
    update_interval: 15s
    accuracy_decimals: 3
    attenuation: 11db
    raw: true
    filters:
      - multiply: 0.00173913 # 2300 -> 4, for attenuation 11db, based on Olivier's code
      - exponential_moving_average:
          alpha: 0.2
          send_every: 2
      - delta: 0.002

binary_sensor:

  - platform: gpio
    pin:
      number: GPIO19
      inverted: true
      mode:
        input: true
        pullup: true
    name: Luxe Volume Up
    on_click:
      then:
        - media_player.volume_up:

  - platform: gpio
    pin:
      number: GPIO32
      inverted: true
      mode:
        input: true
        pullup: true
    name: Luxe Volume Down
    on_click:
      then:
        - media_player.volume_down:

  - platform: gpio
    pin:
      number: GPIO12
      inverted: true
      mode:
        input: true
        output: false
        open_drain: false
        pullup: false
        pulldown: false
      drive_strength: 20.0
    name: Luxe Action Button
    disabled_by_default: false
    entity_category: diagnostic
    id: middle_button
    on_press:
    - then:
      - voice_assistant.start:
    on_release:
    - then:
      - voice_assistant.stop:

light:
  - platform: neopixelbus
    id: top_led
    name: Luxe Top LED
    internal: True
    pin: GPIO22
    default_transition_length: 0s
    variant: SK6812
    num_leds: 1
    type: GRB
    restore_mode: ALWAYS_OFF
    gamma_correct: 2.8
    flash_transition_length: 0s
    effects:
      - pulse:
          name: "Fast Pulse"
          transition_length: 0.1s
          update_interval: 0.2s
      - pulse:
          name: "Slow Pulse"
          transition_length: 0.5s
          update_interval: 1s

Ideas appreciated.

mroxso · May 25, 2023, 7:45am

thats exactly where i am right now as well. i setup everything yesterday and most works.
only the audio response is not working. i didnt try the speaker component.

Deco · May 25, 2023, 11:04am

this worked for me on proto when i tried 2 days ago, it’s using the speaker component and talks back loud and clear, i did however have to change the button pin because my button on proto seems dead but it’s changed back to normal in this code.

Deco · May 25, 2023, 12:28pm

not sure what the goal is, maybe to use mediaplayer but when i tried that did not work, then came the speaker component and that did!, using it on 4 devices now (Muse Proto, Muse Luxe, ESP32Rover and ESP32-D1 mini, all works just fine with the speaker component, i do hope the plan is to keep it and have to ways two output audio

anyway i i have both Ryan (piper) and Google speaking together in some automations, like google saying how cold the cooler is when i stand in front it it, and ryan adding “yeah and outside in the sun it’s …” - great fun! (but i noticed Ryan (piper) speaks louder than google)

gatonero · July 26, 2023, 10:33pm

It would be nice to have a clear step-by-step guide how to get the several devices working including voice assistant. Here are to much discussions between developers for average users.

jkms · September 7, 2023, 4:43pm

Any idea why I might be getting timeouts?

substitutions:
  name: raspiaudio-muse-luxe-39ab54
packages:
  raspiaudio.muse-luxe: github://esphome/media-players/raspiaudio-muse-luxe.yaml@main
esphome:
  name: ${name}
  name_add_mac_suffix: false
api:
  encryption:
    key: <REDACTED>

web_server:
  port: 80

wifi:
#  ssid: !secret wifi_ssid
  ssid: <REDACTED>
  password: <REDACTED>
  domain: <REDACTED>

09:41:28	[D]	[voice_assistant:132]	
Requesting start...
09:41:28	[D]	[voice_assistant:111]	
Starting...
09:41:28	[D]	[voice_assistant:154]	
Assist Pipeline running
09:41:28	[D]	[light:036]	
'RaspiAudio Muse Luxe' Setting:
09:41:28	[D]	[light:047]	
  State: ON
09:41:28	[D]	[light:059]	
  Red: 0%, Green: 0%, Blue: 100%
09:41:28	[D]	[light:085]	
  Transition length: 1.0s
09:41:31	[D]	[binary_sensor:036]	
'Action': Sending state OFF
09:41:31	[D]	[voice_assistant:144]	
Signaling stop...
09:41:40	[D]	[sensor:094]	
'Battery': Sending state 4.03644 V with 3 decimals of accuracy
09:42:00	[D]	[voice_assistant:218]	
Assist Pipeline ended
09:42:00	[E]	[voice_assistant:231]	
Error: pipeline-timeout - Pipeline timeout
09:42:00	[D]	[voice_assistant:144]	
Signaling stop...
09:42:00	[D]	[light:036]	
'RaspiAudio Muse Luxe' Setting:
09:42:00	[D]	[light:059]	
  Red: 100%, Green: 0%, Blue: 0%
09:42:00	[D]	[light:085]	
  Transition length: 1.0s
09:42:00	[D]	[light:085]	
  Transition length: 1.0s
09:42:01	[D]	[light:036]	
'RaspiAudio Muse Luxe' Setting:
09:42:01	[D]	[light:047]	
  State: OFF
09:42:01	[D]	[light:085]	
  Transition length: 1.0s
09:42:01	[D]	[light:036]	
'RaspiAudio Muse Luxe' Setting:
09:42:01	[D]	[light:085]	
  Transition length: 1.0s
09:42:10	[D]	[sensor:094]

Home Assistant shows the following error:

Logger: homeassistant.components.esphome.voice_assistant
Source: components/esphome/voice_assistant.py:230
Integration: ESPHome (documentation, issues)
First occurred: 9:27:45 AM (4 occurrences)
Last logged: 9:41:29 AM

Received unknown pipeline event type: stt-vad-start

s00500 · October 13, 2023, 9:52am

Hey everyone, Year of the Voice part 4 is out and it works for me on the Muse Proto!

BUT

As again I could not get it to work with the config provides by Jesse on the esphome repo. It would work nicely, but the Speaker was dead again. So I modified the config with the wakeword back to use the mediaplayer… this works, but it needs framework arduino and that kills the idf vocie activity detection… so not a preferred solution…

Anyone has any ideas why this only works with the media player and not the speaker component ? I have not yet had more time to investigate…

Zortan · October 13, 2023, 11:57am

Hi s00500

not sure what I did differently than you but the config that I was modifying yesterday uses the speaker component and appears to work (although I’m having trouble getting it to actually recognize the wake word). it’s for the Muse Luxe but maybe it will help you to debug what is going on in your setup

s00500 · October 13, 2023, 8:09pm

Hey @Zortan

Thanks for the comment, yes I was using the freshly modified config, but the one for the proto

But I figured out what the difference is: the mute pin needs to be properly implemented, I have opened a PR to change the config =D, check my changes here

github.com/esphome/issues

Raspi Audio Muse Proto Speaker component not working

opened 09:59AM - 13 Oct 23 UTC

s00500

### The problem Hey all, year of the voice part 4 is super cool! I could not… resist updating to latest beta yesterday and testing this on my muse proto And it does work nicely, but I go t no sound from the speaker using the config found here: https://github.com/esphome/firmware/blob/main/voice-assistant/raspiaudio-muse-proto.yaml So I changed the config to use media_player component instead, this has worked for me in the past, and again it did the trick, this way I get audio out working fine. BUT using media_player requires me to use framework:arduino and therefore I had to disable the IDFs VAD function. Does anyone have an Idea why the speaker does only work when used via mediaplayer ? To be clear: I am mainly reporting that the config in the esphome repo does NOT have working sound output on the Muse Proto. The added example yaml is my modified config that works fine (without VAD) ### Which version of ESPHome has the issue? 2023.10.0b1 ### What type of installation are you using? Home Assistant Add-on ### Which version of Home Assistant has the issue? 2023.10.2 ### What platform are you using? ESP32 ### Board RaspiAudio Muse Proto ### Component causing the issue speaker ### Example YAML snippet ```yaml --- esphome: name: muse1 friendly_name: Muse1 project: name: raspiaudio.muse-proto-voice-assistant version: "1.0" min_version: 2023.10.0b1 on_boot: - priority: -100 then: - wait_until: api.connected - delay: 1s - if: condition: switch.is_on: use_wake_word then: - voice_assistant.start_continuous: esp32: board: esp-wrover-kit framework: type: arduino wifi: ssid: !secret wifi_ssid password: !secret wifi_password fast_connect: true logger: api: encryption: key: ota: password: #external_components: # - source: github://pr#5230 # components: # - esp_adf # refresh: 0s i2s_audio: - i2s_lrclk_pin: GPIO25 i2s_bclk_pin: GPIO5 microphone: - platform: i2s_audio id: board_microphone channel: left i2s_din_pin: GPIO35 adc_type: external pdm: false # speaker: # - platform: i2s_audio # id: board_speaker # dac_type: external # i2s_dout_pin: GPIO26 # mode: mono #esp_adf: media_player: - platform: i2s_audio id: media_out name: Muse1 dac_type: external i2s_dout_pin: GPIO26 mode: mono mute_pin: number: GPIO21 inverted: true voice_assistant: id: va microphone: board_microphone # speaker: board_speaker use_wake_word: true noise_suppression_level: 2 auto_gain: 31dBFS volume_multiplier: 2.0 # vad_threshold: 3 on_listening: - media_player.toggle: media_out - light.turn_on: id: led blue: 100% red: 0% green: 0% brightness: 100% effect: pulse on_tts_start: - light.turn_on: id: led blue: 0% red: 0% green: 100% brightness: 100% effect: pulse on_tts_end: - media_player.play_media: !lambda return x; on_end: - delay: 100ms # - wait_until: # not: # speaker.is_playing: - script.execute: reset_led on_error: - light.turn_on: id: led blue: 0% red: 100% green: 0% brightness: 100% effect: none - delay: 1s - script.execute: reset_led - script.wait: reset_led - lambda: |- if (code == "wake-provider-missing" || code == "wake-engine-missing") { id(use_wake_word).turn_off(); } binary_sensor: - platform: gpio pin: number: GPIO0 inverted: true mode: input: true pullup: true name: Action disabled_by_default: true on_click: - if: condition: switch.is_off: use_wake_word then: - if: condition: voice_assistant.is_running then: - voice_assistant.stop: - script.execute: reset_led else: - voice_assistant.start: else: - voice_assistant.stop - delay: 1s - script.execute: reset_led - script.wait: reset_led - voice_assistant.start_continuous: light: - platform: esp32_rmt_led_strip rmt_channel: 0 name: None id: led disabled_by_default: true pin: GPIO22 chipset: SK6812 num_leds: 1 rgb_order: grb effects: - pulse: transition_length: 250ms update_interval: 250ms script: - id: reset_led then: - if: condition: switch.is_on: use_wake_word then: - light.turn_on: id: led blue: 100% red: 100% green: 0% brightness: 100% effect: none else: - light.turn_off: led switch: - platform: template name: Use wake word id: use_wake_word optimistic: true restore_mode: RESTORE_DEFAULT_ON entity_category: config on_turn_on: - lambda: id(va).set_use_wake_word(true); - if: condition: not: - voice_assistant.is_running then: - voice_assistant.start_continuous - script.execute: reset_led on_turn_off: - voice_assistant.stop - lambda: id(va).set_use_wake_word(false); - script.execute: reset_led ``` ### Anything in the logs that might be useful for us? _No response_ ### Additional information _No response_

This works really well now, I am using openWakeWord on my RPI4 homeassistant install

This is all finally doing what it should, what a dream =D

coderanger · October 13, 2023, 9:07pm

This works for me now! I manually updated my YAML with your additions and now it talks back to me. Initially I got a compile error after I added the PR changes, but once I did a clean build it compiled and installed fine.

coderanger · October 14, 2023, 3:13pm

I was having an issue where I was getting crackling on the speaker on device reboot. There is no restore mode for outputs so I added an on boot command to mute the speaker.

  on_boot:
    - priority: 700
      then:
        - output.turn_on: mute_pin

I’m still having an issue where it listens for the wake word after a reflash, but when I physically restart the board (power cycle / reset button) then it does not.

zephirus · October 19, 2023, 1:14am

I have the Muse Luxe and I’m so confused. What should I flash? Media Player or Voice Assistant? What about the yaml file? Is it necessary? Where do I put it. I have to agree with @gatonero, can someone explain how to do this step-by-step? Thanks!

gatonero · October 19, 2023, 9:59am

I made some success and with the new ESPHome and Home assistant releases it’s working much better. When you flash it both Media Player and Voice Assistant will be enabled at the same time. Go to https://esphome.io/projects/?type=media and start the installation from there.

zephirus · October 19, 2023, 10:08pm

What do you mean with “When you flash it both Media Player and Voice Assistant will be enabled at the same time”? I flashed Voice Assistant only in the Muse Luxe, how can I flash both? and btw, how do I flash Voice Assistant in that site? Is not available.

gatonero · October 20, 2023, 9:49pm

Have a look into the file raspiaudio-muse-luxe.yaml. There is an entry for voice_assistant: and media_player: also. Both functions will be installed with that yaml file at the same time. Here are some detailed instructions how to install it ESPHome config - Raspiaudio Muse Luxe as a voice assistant satellite in Home Assistant

zephirus · October 21, 2023, 10:30am

I already tried to install that yaml. Tried again and always get this error:

INFO Resolving IP address of raspiaudio-muse-luxe.local
ERROR Error resolving IP address of raspiaudio-muse-luxe.local. Is it connected to WiFi?
ERROR (If this error persists, please set a static IP address: https://esphome.io/components/wifi.html#manual-ips)
ERROR Error resolving IP address: Error resolving address with mDNS: Did not respond. Maybe the device is offline., [Errno -5] No address associated with hostname

I saw the documentation but nothing works, I can never get past this.

gatonero · October 21, 2023, 11:28am

Your device isn’t connected, neither by cable nor by WiFi. Have a look here how to connect it either by cable (preferred) or WiFi. Getting Started with ESPHome and Home Assistant

zephirus · October 21, 2023, 12:03pm

I’m using that. I successfully flashed the yaml file you posted in your last comment, adding this part to the wifi section:

manual_ip:
    static_ip: 192.168.1.225
    gateway: 192.168.1.1
    subnet: 255.255.255.0

and I get a success message in ESPHome inside HA.

But then I have this:

As you can see, the device is OFFLINE. If I click EDIT, I see the default configuration, not the yaml file you posted.

Btw, we dont’ need to specify the API key in the yaml?

EDIT: the only way to have the Muse Luxe show as ONLINE is to flash the MediaPlayer again using this page: Ready-Made Projects — ESPHome. And even so, this message shows all the time:

muse luxe

so I always have to reset the device pressing the inside of the AUX port.

zephirus · October 21, 2023, 12:39pm

EDIT: I managed to make Voice Assistant to work, not with the custom yaml but by flashing the simple Media Player in this site - Ready-Made Projects — ESPHome.

From what I see, I have to press the PLAY button when I talk and I never get audio back? Is this expected?

Also, I tried to play a simple MP3 file in my local files and when I click PLAY, I get a clicky sound and nothing plays. Is this also the expected behaviour?