Part 1: Smart Home Matter and Thread Deep Dive

October 21, 2023
in Smart Home
Tags:

The modern Smart Home has a host of new standards to enable what is supposed to be a seamless experience across multiple ecosystems. These new standards include Matter and Thread, both of which the likes of Amazon, Google, Apple, Samsung, and Home Assistant support to various degrees.

There’s A LOT of confusion and mystery around these two new standards, and troubleshooting any issues can be hard. In this series of posts I’ll do a deep dive in parts of Matter, Thread, IPv6 addressing and more with the goal of being an educational resource on your smart home journey. 

Part 1: Smart Home Matter and Thread Deep Dive

  • What is Matter?
  • What is Thread?
  • What can you deploy today?
  • The State of Thread Border Routers
  • Home Assistant Users
  • Thread IPv6 Addressing
  • What is Multicast?
  • Discovering Your Matter Devices with mDNS
  • Decoding Home Assistant IPv6 Addresses

Part 2: Smart Home Matter and Thread Deep Dive

  • Exploring IPv6 Routing Details
  • Home Assistant Matter IPv6 Routing Failover

Part 3: Smart Home Matter and Thread Deep Dive

  • Home Assistant Deployment Model for Matter
  • Home Assistant Matter Controller Logs
  • Proxmox IPv6 Tips
  • mDNS Networking Tips
  • The Sad State of Switch/Router Firmware
  • The Sad State of the Linux Kernel IPv6 Routing
  • Ubiquity Suggestions

Yes that is a lot! But this will be a fun journey and you should learn a lot. Even if you aren’t using Home Assistant or Apple TV smart hubs, all of this knowledge will apply to Matter and Thread in general so keep reading. 

December 8, 2023 Update: Massive overhaul of the State of Thread Border Routers section in Part 1. Added a new section dedicated to Home Assistant users. This section includes many recommendations directly from the Home Assistant developer team to help ensure a smooth Thread/Matter experience. Added a new diagram to the Matter section, courtesy of Google. 

December 3, 2023 Update: Added a section in Part 3 for some suggested Ubiquity settings to try. 

November 28, 2023 Update: I modified a couple of sections in Part 1 to better clarify Thread border router options. I also clarified the state for Home Assistant users. I added a new section called “What can you deploy today?”. 

November 13, 2023 Update: Updated the Thread border router section and recommendations. Google Nest and Eero routers now support Thread v1.3 with TREL, joining Apple. That makes all three border routers solid options.

October 29, 2023 Update: I added two sections to Part 3 covering the sad state of Switch firmware, and the sad state of Linux IPv6 routing. Most of the information came directly from Home Assistant developers for Matter and Thread. Lots of good information!

What is Matter?

Matter is an open-source, unified IP-based connectivity protocol built for smart home devices, backed by major companies like Apple, Google, Amazon, and the CSA (Connectivity Standards Alliance). Matter aims to increase compatibility among smart home devices and increase the security of these devices. Key features of Matter include:
 
  • Local Unified IP-based connectivity: Devices from different manufacturers can communicate directly with each other (without the cloud–all locally) using this standard IPv6-based protocol over common networking technologies like Ethernet, Wi-Fi and Thread. It does NOT support z-wave or Zigbee. Bluetooth is ONLY used for commissioning a device, not for ongoing communications. 
  • Security: Matter focuses heavily on security, requiring device-to-device encrypted communications. This includes secure setup codes to prevent unauthorized access. Matter security is superior to z-wave security, as Matter requires a minimum (good) security baseline, device attestation, and always uses encrypted communications. 
  • Ease of use: Matter devices use a simple setup process, enabling consumers to easily add and control devices within a smart home environment, across manufacturers. Apple donated much of the Homekit protocol, which was used as the basis for Matter commissioning. 
  • Compatibility: Matter is designed to support a wide range of device types, from lighting and electrical products to smart home controls, access control, security, and more. The Matter v1.2 spec adds a number of new device types and is now public. You can read the Matter 1.2 press release. 
The graphic below shows you how all of the layers of the Matter stack line up with the standard OSI model. For additional information, check out the Google Matter Primer. 
Image courtesy Google

Bottom Line: Matter is a SOFTWARE control plane protocol (NOT a networking protocol) for your smart home. It’s built on IPv6 and is independent of your underlying network protocols, like Thread or WiFi. Apple, Google, Amazon, Samsung, Home Assistant, etc. all have Matter controllers. Matter devices, unlike z-wave or Zigbee, must support multiple simultaneous controllers. So, for example, Home Assistant and Apple Home can both directly control a Matter device and the device’s state is reflected in real time to both controllers. This cross-vendor multi-controller support is new in the smart home space and very exciting. 

What is Thread?

Thread is an IPv6-based, low-power, secure, and interoperable radio frequency mesh networking technology specifically designed for communicating with and among devices within a smart home environment. It’s designed to overcome some limitations of older technologies to provide a more reliable wireless communication protocol especially for home automation. Thread has been around for several years. Apple home hubs have used it for some HomeKit devices. 
 
Key features of Thread include:
 
  • Reliable – Thread networks are self-healing. It supports mesh networking, which means each device can act as a node that can relay data for other devices. The more devices, the stronger the network. 
  • Standards based – It’s IP-based (IPv6), facilitating direct addressability of devices on the network from the Internet. Thread runs on 802.15.4 radio hardware, the same protocol used by Zigbee. Some newer 802.15.4 radios can be flashed to support Thread, and some can even do Zigbee and Thread at the same time.
  • Secure – End-to-end secure and encrypted communications are a baseline requirement. This is different from Z-wave where security is optional and adds additional overhead on your Z-wave network. 
  • Scalable – Thread networks are designed to scale to hundreds of devices. 
  • Power Efficiency – Low-power thread device can be battery powered and can operate for years. 
A Thread Border Router (TBR) is a device that provides connectivity from the Thread network, which is a low-power wireless device network, to other more traditional networks, typically IP-based networks like WiFi or Ethernet. A TBR has several important functions:
 
  • Gateway to the Internet: It connects Thread devices to your local LAN, and by extension the Internet. The TBR acts as the bridge between these two kinds of networks, allowing Thread devices to access Internet-based services.
  • Network Configuration and Management: A TBR controls the formation of the Thread  network, determining which devices can join the network, enforcing security policies, and routing data between devices. A TBR also determines IPv6 prefixes and routes for the Thread and adjacent infrastructure networks so that devices on both sides of the network can communicate. 
  • Mesh Network Formation: As part of the Thread network, a TBR can also help to form and extend the mesh network topology, relaying communications between Thread devices that can’t directly communicate due to distance or obstructions.
  • mDNS Publisher: A TBD uses multicast DNS (mDNS) to publish mDNS DNS-SD discovery packets on behalf of the thread nodes so they can be discovered on adjacent networks. 
  • Channel Management: A TBR is also in charge of which Thread RF channel the mesh network uses. This is very similar to the role of a Zigbee controller, which determined which Zigbee channel the network uses.  

Some smart home hubs, such as Apple TVs on tvOS 17 release support TREL. TREL allows any Thread border router that has other IP-based links (e.g. WiFi or Ethernet) to incorporate it into the Thread mesh topology. This allows the Thread network to leverage the benefits of the other links (e.g. higher throughout, capacity, better coverage etc.) and keep it more connected. It also can help reduce traffic over the Thread radios by allowing border routers to talk to each other over their higher capacity WiFi/Ethernet links.

The latest version of Thread is v1.3 and it was released to the public on February 27, 2023. Smart home hub manufacturers such as Apple, Google, Amazon and Samsung often include Thread radios in their devices. You can read more about Thread here. Thread 1.3.1 will make TREL mandatory, which is great news.

What can you deploy today?

Despite some of the messiness of Thread and Matter in their current state, today you can build a rock solid and very responsive smart home with Thread and Matter components. Do all device types come in Thread/Matter variants? Absolutely not. Thus, it’s unlikely you can exclusively use Thread/Matter devices. You can easily use other protocols in your smart home for other devices, such as Zigbee or even z-wave. A great feature of Home Assistant is that you can use multiple smart home protocols, pretty much seamlessly. If purchasing a new IoT device today, would first consider the thread/matter version if it exists. If so, go that route. If not, then pick a fallback protocol like Zigbee. 

Follow this simple recipe for the best Thread and Matter experience: 

  • Use a Thread border router from Apple or Google. $99 USD entry point. 
  • Deploy Home Assistant OS (HAOS) – Not Docker, not supervised.
  • Ensure your network gear doesn’t have broken IPv6 multicast support.
  • Ensure all of your IoT devices and ALL smart hubs/controllers, like Home Assistant, are on the same VLAN. The VLAN can be isolated, with firewalls ports open into the IoT VLAN as needed.  

Each of these points are covered in detail in this series. Matter enabled Thread devices should have a nearly instant response time in all of your Matter controllers, like Home Assistant. It should not be laggy or unreliable. 

There are some pain points with Thread and Matter today, so it’s not all roses or perfection. Solutions are still maturing. But I have over 20 Thread/Matter devices on my network with Home Assistant and Apple Home. Response times are nearly instant and the network is highly reliable. But it took a bit of work to get there (more on that in Part 3). This series will help you avoid the pitfalls I ran into and deploy a highly reliable and responsive smart home from the get go. 

The State of Thread Border Routers

WiFi APs and Thread Border Routers

Think of Thread (remember it’s a Radio Frequency protocol), in the same general terms as your WiFi network. What do I mean by that? When you deploy WiFi in your home 99% of people will buy commercial WiFi access points or routers and use the same brand if they have multiple APs. For example, people will buy WiFi APs from Google, Netgear, TP-Link, Eero, Asus, Ruckus, Ubiquiti, etc. And if you want a mesh WiFi system, you use multiple APs from the same vendor that are designed to work together as a mesh. 

Bottom Line: Consumers put their faith in commercial WiFi APs to provide a polished, secure, standards based stable solution. Thread is basically in the same boat. A commercial Thread Border router that supports Thread 1.3 and TREL is your best option for a production ready Thread Network. Specific recommendations are below. 

Thread Islands and Credential Sharing

Since many vendors are now including Thread Border router functionality in their products (Google, Amazon, Apple, Nanoleaf, Samsung, Eero, etc.), it will not be unusual for your home to have multiple Thread border routers. And since there is not yet an industry standard for Thread credential sharing, these border routers will be little “islands.” 

Just like WiFi, it’s best to have your Thread devices on a single “SSID” to avoid confusion and complexity. This can easily be accomplished by always provisioning your Thread devices to the same Thread Border Router ecosystem. 

Today, both Apple and Google have public Thread Credential sharing APIs that platforms like Home Assistant can take advantage of. In fact, as of the iOS Home Assistant companion app 2023.490, Home Assistant can now import your Apple Home Thread credentials. This has very limited utility today, but I’ll cover this in the Home Assistant section below. 

The Thread group wants to eventually publish a standard for Thread credential sharing. Thread credential sharing will propel Thread past WiFi in terms of vendor interoperability, and allow Thread Border routers from Apple, Google, Amazon, HA/Skyconnect, etc. to all use the same Thread network. However, there are still complexities such as which border router will manage the other aspects of Thread such as RF channel, IPv6 addressing, etc. So credential sharing is only one piece of the interoperability pie. The Thread working group wants to address all of these issues over time. 

The best advice is to pick a single vendor (e.g. Google, Apple, etc.) and always commission your new Thread devices to that Thread Border router ecosystem. This ensures that all of your Thread devices are on the same Thread network. 

Thread Border Router Recommendations

The general recommendations for choosing your Thread Border router are easy:

  • If the lead smart home nerd has an iPhone, use Apple Thread border routers.
  • If the lead smart home nerd has an Android phone, use Google Thread border routers. 

Check out this link for more information on which Apple devices are Thread Border routers. Check out this link for information on which Google devices are Thread border routers. Both platforms offer a $99 USD entry point for a Thread border router. For redundancy, multiple Thread border routers from the same vendor are encouraged. 

December 2023 Thread Border router state:

  • Thread 1.3 with TREL: Google, Apple, Eero
  • Legacy Thread: Amazon Echo, Smarthings, Home Assistant

It doesn’t matter if other members of the household use a different phone ecosystem. All that is important is what phone OS the person who will be adding Thread devices to the home automation platform of choice. 

Home Assistant Users

Supported Scenarios

As of October 2023, the following are the supported Home Assistant scenarios. These bullet points are a direct copy and paste from the Home Assistant Matter Discord pinned post by Marcel (HA Matter developer from Nabu Casa):

  • Use the HA Companion app on iPhone or Android to commission a Wi-Fi based Matter device (using the phone to do the bluetooth commissioning to the device).
  • Use the HA Companion app on iPhone to commission a Thread based Matter device utilizing existing Apple Border router(s) like Homepod or ATV 4K.
  • Use the HA Companion app on Android to commission a Thread based Matter device utilizing existing Google Border router(s) like Nest Hub V2 or Nest WiFi Pro.
  • Receiving a shared device (existing Matter device in Apple Home, Google Home, Smartthings etc.) in HA: Supported but you need the HA Companion app as well for the time being.
  • Commission a Matter bridge device to HA (e.g. Aqara or Switchbot Hub): Supported but you need the HA Companion app as well for the time being.
  • Using Thread border routers from another vendor (so, not Apple or Google) with Home Assistant is possible but a bit more cumbersome. If you have the thread dataset (credentials) you can paste those in the Thread config panel of Home Assistant and use the Android app to commission a device (so not iOS). A more convenient way is if you can commission the device to that platform itself and then share it from there to HA.

Home Assistant Thread Border Router

Home Assistant users are often concerned about privacy and local control of devices. This is very understandable, and can be a big reason why users choose Home Assistant over other smart home automation platforms. To again quote Marcel from Nabu Casa:

A common misconception seems to be that using existing Apple/Google border routers is then less privacy friendly but that is not true as we just leverage the Thread network but our Matter controller creates its own trust relationship with the devices. No need to pair devices to Apple or Google, we just use their network the same way we can communicate over WiFi based Matter devices using access points from various vendors.

As Marcel points out, today you connect many smart home devices to your WiFi network, which is from a commercial vendor. Thread is no different. 

The Home Assistant team is actively working on Thread credential sharing with commercial Thread border router vendors, like Apple and Google. As I mentioned before, iOS Home Assistant Companion app 2023.490 (beta) can import Apple Thread credentials. I’m sure Google Thread Credential importing is not far behind. 

However, the Thread Border Router software functionality in Home Assistant is still under development and not ready for prime time. To quote another pinned post on Discord from Marcel:

Using the Home Assistant Yellow or SkyConnect as Thread Border router is not yet entirely production ready so we’re not yet advertising it and we’re missing a few building blocks.

Yes, over time the Thread Border router functionality in Home Assistant will mature and support TREL. But as of December 2023, that is not the case. Even though today you can import Apple Thread credentials, that’s not at all useful until the HA Thread Border Router supports TREL.

Matter Controllers

The Matter controller in Home Assistant works very well with the Thread border routers from Apple and Google. Home Assistant establishes its own trust relationship with each Thread enabled Matter device. You do NOT need to pair your Thread device to that ecosystem’s smart home controller application. This means there is ZERO requirement to have the Apple or Google Home apps even be aware of any Matter devices or control them. Just like WiFi, we are only using the commercial Thread border routers to manage the RF network. 

To be clear, this means that when Home Assistant wants to control your Matter enabled smart plug, for example, the HA Matter controller will send the encrypted command directly to the IPv6 address of your Thread device and the Thread device will send its status directly back to Home Assistant. The Apple or Google Matter controller is NOT involved. The Apple/Google Thread border router is merely acting as a gateway to the Thread RF network.  

However, if you want Matter controller redundancy, then you can optionally use the multi-admin feature of Matter and also pair your Matter devices to other ecosystems. For example, even though I use Home Assistant as my primary smart home platform, I also pair all Matter devices to Apple Home. Why? In case HA is down then I have a backup means to control devices. 

Thread IPv6 Addressing

Understanding Thread device IPv6 addressing is important. As we dive deeper into Thread this will become apparent. Thread device will have multiple IPv6 addresses assigned to them. One category of addresses are called Endpoint Identifiers (EIDs), which identify a unique Thread interface within a Thread network partition. EIDs are independent of Thread network topology. A few Thread unicast EIDs include:

  • Link-Local Address (LLA): An EID that identifies a Thread interface reachable by a single radio transmission. Always has a prefix of fe80::/16.
  • Mesh-Local EID (ML-EID): An EID that identifies a Thread interface, independent of network topology. Used to reach a Thread interface within the same Thread partition. Also called a Unique Local Address (ULA). Always has a prefix fd00::/8.
  • Global Unicast Address (GUA): An EID that identifies a Thread interface on a global scope, beyond a Thread network. These prefixes can vary depending on how your smart home hub wants to do addressing. This is the address which the rest of your home LAN sees for your thread devices. 

Later in this post we will see how to discover those IPv6 addresses for each Matter enabled Thread device in Home Assistant. But just to give you a sneak peek, here’s an example for a Thread device:

text

What is Multicast?

Understanding multicast is key to understanding Matter. Matter relies on IPv6 mDNS (multicast DNS) to advertise services. If multicast is somehow broken on your network, you will likely have BIG random Matter headaches. 

Multicast is a method used in the Internet Protocol (IP) to transmit data packets to multiple destinations simultaneously. It uses efficient network routing and minimizes bandwidth usage since a single packet can be received by multiple recipients, rather than sending individual packets to each recipient. Multicast uses UDP. UDP is connectionless and does not reorder packets or request retransmission of lost packets, thus making it suitable for multicast delivery where delivery speed is more important than reliability.

IGMP and MLD snooping are techniques that network devices can use to conserve bandwidth by reducing multicast traffic. Both IGMP and MLD snooping improve the performance of Ethernet networks by limiting the extent of the multicast domain and decreasing unnecessary traffic. The Internet Group Management Protocol (IGMP) is a communications protocol used by hosts and adjacent routers on an IPv4 network to establish multicast group memberships. It allows a computer to tell its local router that it wants to receive multicast traffic for a specific group.

Multicast Listener Discovery (MLD) is a component of the Internet Protocol version 6 (IPv6) suite. MLD is derived from IGMP; MLD version 1 (MLDv1) is equivalent to IGMPv2, and MLD version 2 (MLDv2) is equivalent to IGMPv3. MLD is a subprotocol of Internet Control Message Protocol version 6 (ICMPv6), and MLD messages are a subset of ICMPv6 messages.

Apple Home Hub Setup

On my network I have two Apple TV 4Ks that have both Thread and Ethernet. This means I have two Thread border routers on my network. Both run tvOS 17.1. Apple designates one home hub as “Connected” and all others as “Standby“.  The “Connected” home hub is the one which advertises the Matter controller services, although both are active v1.3 Thread Border routers with TREL support. But let’s not get ahead of ourselves..we are only getting started. 

graphical user interface, application

Discovering Your Matter Devices with mDNS

If you have an iPhone or Mac you can discover your mDNS enabled devices, including Matter, by using the app Flame.  Simply install Flame on your iPhone or Mac, launch it, and wait a few seconds for the list to populate. Depending on how many devices you have on your network, you may see a long list that includes printers, NAS, network switches, receivers, etc. There is another iPhone/Mac app called Discovery – DNS-SD Browser that presents the same mDNS data but in a different format. However, it has no data export feature. But its tree view can come in handy. 

To find all Matter enabled devices in Flame you can simply type Matter in the search box. To easily view all of the data, you can use the share icon in the upper right corner and airdrop the JSON file to your Mac from your iPhone. Or if you are running it on your Mac, it can directly save a JSON file. I suggest opening the JSON file in a code editor, such as Microsoft Visual Code Studio. 

In the Flame output you can see the IPv6 address of each device and the 16 character hostname. This 16 character hostname is the hex code version of the randomly generated extended MAC address of the thread radio after commissioning is complete. 

In my case the global IPv6 address space that all of my Matter devices are using is prefixed with fd5d:443b:99ef. This is known as the off-mesh routable (OMR) prefix. This prefix is not universal, so your Thread network will likely have a different prefix. Thread devices are assigned two more IPv6 addresses: link-local and mesh-local. Flame will not show you the other two IPv6 address of a Matter device, but as mentioned before, Home Assistant can provide this information. 

graphical user interface, text, application

In the figure below if you look at the two Apple TVs listed on the left, you see one has 7 services and the other has 8. The one with 8 is the “Connected” Apple TV which is your active Matter controller. The other Apple TV doesn’t broadcast the “Matter” service as it’s in standby. However, both Apple TVs do broadcast as a thread border router with TREL support (_meshcorp._udp and _trel._udp, respectively). So for IPv6 packets going to your Thread devices, both Apple TVs can route packets in an active/active fashion. If the network connection to the Loft ATV gets interrupted, the living room ATV will take over as the Matter controller and advertise the _matter._tcp service. 

graphical user interface
graphical user interface, application

If you take a look at the Flame JSON output for an Apple TV we can see some interesting fields. Super nerdy so won’t cover here, but take a gander. 

timeline

Decoding Home Assistant IPv6 Addresses

As of HA 2023.10, finding the IPv6 addresses of your Matter devices in Home Assistant needs a bit of work. Unfortunately this is not yet surfaced in the Home Assistant UI, but it is coming pretty soon. But in the mean time, lets use the manual method. In the Matter diagnostics file for a device the IPv6 addresses are base-64 encoded and we can easily decode them. Let’s get the device’s JSON diagnostics file:

Settings -> Devices & services -> Matter -> Click on xx Devices -> Click on any device -> DOWNLOAD DIAGNOSTICS. 

graphical user interface, application

Open the JSON file and search for IPv6Add and you should see a block of base-64 encoded strings between quotes that start with a / and end with ==. In my case I see three blocks of text. 

text

To decode those base-64 strings into a hex string formatted like an IPv6 address, you can use the following Python script (courtesy of ChatGPT). Just replace the base-64 strings on lines 4-6 with your strings. Run the script, and it should output a nicely formatted IPv6 address. If you don’t mind manually putting in colons, you can use this website to do the base64 to hex conversion: Base64Guru. The Python code can easily be run without any fuss with VS Code (free). 

				
					import base64

base64_strings = [
"/oAAAAAAAAAUwgnlaZDaPg==",
"/RysaSl2AADWZBzKf93Rtg==",
"/V1EO5nvAAAEYQB6xk26Yg=="
]

for base64_str in base64_strings:
    # Decode the Base64 string to bytes
    bytes_arr = base64.b64decode(base64_str)

    # Convert the bytes to a Hex string
    hex_str = ''.join(f'{byte:02x}' for byte in bytes_arr)

    # arrange the Hex code into the IPv6 format
    ipv6_str = ':'.join(hex_str[i:i + 4] for i in range(0, len(hex_str), 4)) 

    print(ipv6_str)

The image below is the Python script output of the Matter device IPv6 addresses. You will recognize the fdf5:44eb:99ef: prefix is the one shown in Flame. So we know this is the global IPv6 address that we can use to ping the Thread device. 

Take note that sometimes Apple assigns a new IPv6 address to Thread devices. So if you ping a device and it’s not responding, double check you have the right IPv6 address..it may have changed. While I was doing some Apple TV routing failover tvOS 17 decided to get a brand new Thread IPv6 GBA, so all devices had a brand new IPv6 prefix. As of HAOS 2023.10, the diagnostics file does NOT get regularly updated with the new IPv6 address. So until that is addressed (coming soon), use my next method below.

There is an alternate way to also find the IPv6 addresses of your Home Assistant Thread devices. The best UI for this is the Discovery DNS-SD browser I mentioned earlier. In the long list of Matter devices it will show you, there will be a block of addresses where the second half of the name is all zeros except for a couple of digits at the end (depending on the number of devices you have). Here, the hex encoded numbers correspond to the “node ID” (in decimal) that appears in the Home Assistant Matter logs and diagnostics file. Although hard to see, but you can expand the mDNS entry and see the faint IPv6 address. 

For example, hex 33 (the last entry in the red boxed area), is 51 decimal. If I dump all the diagnostics files for all of my Matter devices, I can see node 51 is one of my window shades. 

Summary

In this post we covered what is Matter, Thread Networks, Thread IPv6 addressing, multicast, discovering your mDNS services, and how to decode Home Assistant IPv6 addresses. In Part 2 I do a deep dive on Matter IPv6 border router failover with two Apple TVs and Home Assistant. Check it out for some heavy networking knowledge. 

Buy me a coffee

Related Posts

Subscribe
Notify of
0 Comments
Inline Feedbacks
View all comments