Performance and Service Level Verification of 5G private networks

Are we trying to reinvent the wheel?

Manufacturing is already very automated and many ask: “Aren´t existing manufacturing processes good enough?” The current demand for transforming manufacturing processes we now see mainly comes from the business side. There are limits to the cost reductions possible using standard approaches. Many consultancies base their business models on delivering cost-cutting projects to customers, where they examine the entire production chain to identify potential for cutting operating costs. Their measures are struggling to produce the expected reductions and cost pressures continue to rise, making innovative approaches necessary.

5G technologies address this challenge at its source for faster and safer operations, while creating new capabilities in industrial processes. This approach allows factories to connect various facility components over 5G networks, including manufacturing robots, AGVs or other transport vehicles. They can be managed more easily, and production lines can be quickly refitted to shorten lead times. 5G can also replace existing WLAN networks in production facilities to overcome WLAN limitations for seamless mobility, stronger security, greater speed and efficiency in the manufacturing process and decreased operating costs.

Coverage and performance testing

Keeping in mind all these requirements, a network must provide the best from the entire 5G triangle:

1) mMTC – helps networks handle many connections in parallel when there is a high number of connected devices
2) eMBB – enhanced mobile broadband helps when high broadband and high throughput are required (e.g. remote video inspection via 5G at Lufthansa in Hamburg)
3) URLLC – ultra reliability ultra-low latency services (e.g. low-latency based robot communication)

As our series has already discussed, private networks are implemented in five phases. Engineering, rollout preparation, site acceptance, performance & service level verification and troubleshooting.

Solutions from Rohde & Schwarz mobile network testing have been in use since the creation of the very first 5G engineering use cases and pre-commercial trials of 5G private networks. Our 5G scanners (R&S®TSME6 and R&S®TSMA6B) in combination with our drive test software ROMES or R&S®QualiPoc Android have given us the lead in 5G network testing. The setup enables passive testing. Simply by providing the wanted bands for scanning, the setup automatically identifies on-air carriers and their operators, decodes master information blocks (MIB) and system information blocks (SIB). The information makes it easy to find broadcast details for every on-air operator or measure each beam and cell.

However, to completely understand the quality of a 5G network used to connect robots or AGVs, active measurements with a 5G device are needed. These measurements let you view performance from the user perspective and verify the uplink access behaviour, mobility procedures, layer 3 communications between the device and network, as well as the QoE and QoS at the application level.

The Rohde & Schwarz answer to private network testing challenges

Private networks are mostly standalone and implemented in a dedicated band, currently usually the Sub6 GHz band. They have their own MNC and MCC coding and specific SIMs that are used to connect to the network.

Passive (scanner) measurements in private networks

Industry environments (such as production lines with many robots) have very harsh multi-path radio propagation with high interference levels from various machines and robots. Our scanners can properly test many of these challenges in advance:

• Automatic channel detection (ACD) can detect all on-air channels with their bandwidths. MIB and SIB decoding provides insights into the broadcasted information (such as TDD patterns).
• Coverage and beam verification where beams are measured with one scanner for comparing the actual radiated beam pattern with the planned one.
• Time synchronization in TDD networks is important to prevent interference between symbols in an unsynchronized network. Time of arrival (ToA) measurements can help detect broken or unsynchronized components.
• Power spectrum measurements provide insights into the spectrum and can help identify potential interferers in the used band. They show the quality of network filters, since private network spectra are often close to public ones.

Active measurements in private networks - testing approach

• Network connectivity of many connected devices is tested with a ping test
• URLLC of 5G private networks are tested with an interactivity test
• eMBB (Network throughput) is tested with an iperf3 test

Most private networks cannot be accessed from outside. The tests above are best performed using a server we have installed in a private network. This server handles all active measurements (ping, Interactivity testing and iperf3). This allows us to perfectly mimic the real-life scenario of robots or AGVs communicating via 5G to the network in an industrial environment.

Active measurements in private networks for engineering

Connection related details (such as uplink performance, radio access control, mobility control or application layer performance) require an active device with a SIM card. We have integrated the industry-based module Quectel RM520N into our measurement solutions for such scenarios. The Qualcomm x62 based module is on a development board for a USB-C connection and a SIM card slot. This is an unmodified, commercially available module, connected with a USB-C cable to a scanner (above picture). It can measure networks from the user perspective, such as a robot or an AGV.

By complementing the passive measurement setup with a Quectel module, we could control the connected module with ROMES software and configure various tests to verify all major aspects of the 5G triangle.

Active measurements in private networks for acceptance, verification and troubleshooting

Private networks certainly use different PLMN (MCC/MNC) compared to public mobile networks to clearly differentiate. Not all smartphones made for the public mass market provide means to register to private networks with such non-typical MCC/MNC combinations.

By offering the R&S®QualiPoc Android on the Crosscall CORE-Z5 device (that also targets the mission-critical or public safety market), we can confirm that a specific PLMN is supported by this device. In the past with other vendors, we first must verify on-site, whether the device works fine within that private network. Now, the Crosscall saves much time in the process from offering to the first use at the customers network with QualiPoc software.

Another device that can be used for active measurements in these test phases is the R&S®LCM network monitoring probe described in detail in the next paragraph.

Active measurements in private networks for performance monitoring

The performance monitoring phase requires long-term stable and proven test solution and a development board as frontend is not good enough. For this purpose, Rohde & Schwarz developed the R&S®LCM network performance monitoring probe.

R&S®LCM combines a powerful Linux computer with a high-performance 4G/5G IoT/eMBB data module, all within a compact and robust design. The integrated R&S®QualiProbe software, leveraging more than a decade of network performance testing expertise, provides unparalleled insights into service quality measurements, ensuring accurate and reliable data analysis. R&S®LCM is the ideal comprehensive service quality monitoring solution for critical mobile networks, offering a unified view of mobile network performance through local control of a single unit or remote control of multiple units via the SmartMonitor server and collected data analyzed in SmartAnalytics.

SmartMonitor is used to manage remotely the LCM network performance monitoring probe placed in critical and strategic locations. The tool allows users to monitor each probe in detail and in real time, displaying the results on a customizable dashboard. With its event and notification management interface, the user gets immediately informed when an event occurs. It offers multiple options to notify other users or supervisors.

What have we found?

We have been very active in measuring various private networks that have been installed in agricultural scenarios, automotive production and the chemical industry relative to those designed as testbeds at universities or exhibition centers (e.g. the Hannover fairground). Most private networks come from well-known suppliers, such as Nokia or Ericsson. However, we also see OpenRAN growing more popular.

Ping testing verifies a connection and how quickly the network responds. The test sends a 32-byte packet to the server within the private network. The packet sizes can be adjusted with ROMES as needed. Our recent measurements have revealed very good ping values, some even below 10ms. This is still far from the URLLC promise of 1ms latency for 5G, but it is a good sign for the future.

Interactivity tests use predefined patterns against a server installed in the private network. Ping testing is good enough to test end-to-end connectivity. Understanding network performance closely resembling real-life scenarios (such as Industry 4.0 process automation) is possible with our interactivity test that we developed in house based on TWAMP. R&S®ROMES4 provides a long range of various patterns, each of them with a specific:

• number of packets sent per second
• packet size
• packet delay budget
• test duration

We engineer various patterns and offer them for testing appropriate use cases and latency in 5G networks. Here is an example of one measurement campaign that measured various patterns:

The roundtrip times (RTT) for such patterns here are a bit higher than those for pure ping tests. This was expected, since the length of such patterns corresponds with real traffic patterns and should be seen as a more realistic measurement than ping tests.

Every pattern in the table had an extensive metric (RTT, packet delay variation etc.). Users can also zoom into the details and find out the latency per packet, as seen in the chart below. This helps identify network locations with high packet delay for troubleshooting and optimizing them.

Iperf3 measurements were usually within the range of 800Mbps – 1Gbps (tested from the module against a server). The measurement below used the UDP protocol with 10 parallel streams. UDP is the main protocol for real-time applications since it avoids additional, uncontrollable traffic with acknowledgements and retransmissions. We have generally seen UDP protocols deliver higher throughput than TCP ones. Please note here that we used an empty network with no additional users, which needs to be taken into account when evaluating the overall quality of the eMBB.

Here is an example of a TCP vs. UDP iperf3 measurement:

For troubleshooting potential interactivity issues (e.g. high packet delay variation) or throughput problems, R&S®ROMES4 can examine network performance at the TTI level. The tool can store the data from the TTI level to provide very detailed insights into potential issues from multiple network layers. R&S®ROMES4 customers use this feature for troubleshooting throughput drops, network access issues, high block error rates, etc.

Here is an example of the 5G TTI view with detailed insights into RACH failures with a clear overview of retransmissions, modulations, TA, RACH requests etc.

Conclusion

The importance of private networks is growing in various industries, particularly in manufacturing, where reliance on public 5G networks is often not a viable option. To meet the stringent requirements for high reliability, performance, and security, many manufacturers are opting for private 5G networks, which enable them to keep sensitive data on-premises. For instance, when it comes to controlling entire production lines, such as in the automotive sector, manufacturers prefer the security and control offered by private networks over public ones. While our recent measurement campaigns have shown that there is still room for improvement in achieving the desired levels of latency, stability of latency, and throughput in high-traffic networks, the outlook is promising, and we are making progress towards meeting these targets.

Learn more about 5G private network testing and watch our on-demand Webinar