Multiple input, multiple output (MIMO) technology, the foundation for the next generation of Wi-Fi products, leverages multiple transmit and receive antennas to deliver greater wireless throughput and range, enabling ubiquitous high-speed voice, video and data services. Today, three basic methods can be used to test MIMO-enabled devices.

  1. Over-the-air testing (for real-world performance spot check testing).

  2. Controlled RF testing with channel emulation (for performance testing).

  3. Controlled RF testing with static channel (for functional test).Each of these testing methods offers trade-offs in terms of time, cost, reliability, repeatability and automation.

MIMO-enabled device testing challenges

The emerging 802.11n standard, with the presence of up to four transmit (Tx) and up to four receive (Rx) chains, significantly increases the complexity of the physical (PHY) layer in comparison to the 802.11a/b/g standards' much simpler single-input, single-output (SISO) architecture. For example, there are more than 300 modulation-coding schemes (MCS) in the 802.11n standard, which are functions of channel bandwidth, number of spatial streams and types of modulations. Complexity of the 802.11n media access control (MAC) layer has also increased substantially compared to that of 802.11a/b/g technologies.

This new standard has dramatically augmented the number of possible configurations in which products can be architected to operate or more accurately, interoperate, increasing test requirements in several ways.

Ensuring that the MAC and PHY layers operate properly in an 802.11n system requires extensive functional testing in device-to-device and device-to-network configurations. An example is the rate adaptation algorithm, which has more variables and options than before and requires a systematic ability to validate that the algorithm functions properly.

The goal of the MIMO architecture is to provide the highest possible throughput and the longest possible range by optimizing the use of the Tx and Rx chains, the modulation schemes, channel bandwidths, etc. to deliver solutions with maximum performance. Semiconductor and equipment vendors will all work to differentiate their products by improving their algorithm implementations, driving demand for comprehensive performance testing. The performance of each product must be validated in all possible operating modes and in as many different multipath environments as possible.

Finally, given the many potential MIMO device configurations, validating the interoperability of these devices with other 802.11n MIMO devices, as well as with legacy 802.11a/b/g SISO devices, is critical to maintaining the ubiquitous operation of the Wi-Fi network.

The challenge facing engineering teams tasked with developing 802.11n systems is how to conduct sufficient functional, interoperability and performance testing required to deliver a high-quality product in a business environment that demands that products be delivered to market in less time and with less cost.

Over-the-air testing

Real-world over-the-air (OTA) testing is used to test the performance, functionality and interoperability of Wi-Fi devices in the production environment in which they will operate. While this approach is in fact realistic, the result is a manual, slow “spot check” of a single test scenario, which is only representative of the conditions at that moment in time. Examples of performance measurement using OTA testing include basic RF performance (Rx sensitivity, EVM), throughput vs. range, and packet error rate (PER) vs. range.

Although, many silicon vendors and device manufacturers have focused on manual OTA testing for 802.11a/b/g SISO devices, the complex MIMO environment and the requirement for multipath makes this type of testing more difficult and less comprehensive for 802.11n. Manual OTA testing conducted in an uncontrolled RF environment is subjected to random, uncontrollable statistical effects, which leads to irreproducible test results. Non-reproducible test results significantly increase the difficulty of confirming the existence and resolution of MIMO system design problems. When manual OTA testing is conducted in a controlled RF shielded room, the enclosure size limits the ability to test distance and motion typical of a real-world scenario. In addition, the time consuming manual test setup and execution typical of OTA tests combined with the extremely large number of test cases required to validate the performance and interoperability make it nearly impossible to achieve the level of test coverage required to develop a high-quality 802.11n product.

While OTA testing is limited in its ability to conduct comprehensive performance and interoperability testing of MIMO devices, OTA testing can be relied upon to provide a quick “spot check” of the performance of a device during product development and field deployment.

Performance testing

The real-world conditions in which MIMO transmitters and receivers operate are constantly changing due to many variables, including device movement, the environment, the movement of people and cars, etc. To accurately measure and test the performance of MIMO systems in real-world environments, dynamic channel emulation using a MIMO channel emulator is required.

A channel emulator, such as the Azimuth ACE 400NB, uses sophisticated digital signal-processing technology to reliably and accurately recreate, in a controlled lab environment, the channel conditions that occur in real-world wireless transmission. With this capability, dynamic channel emulation using a MIMO channel emulator is the most effective method for comprehensively testing the performance of MIMO products. Channel emulators are programmable devices that emulate multipath impairments by subjecting RF signals to real-world obstructions and reflection conditions. Channel emulators digitize the RF signal from a wireless access point (AP) or client; impair the digitized signal by injecting delay, amplitude and phase variation, fading due to obstructions/movement and duplicate copies of the signal (typically as a result of multipath); and regenerates the impaired signal as an RF signal that can be read by an AP or client. The formula for creating the impaired signal is a “channel model” — a mathematical instantiation of a real-world scenario like a home, office, or an open area like a hotspot in a mall. A channel model is a statistical model — it changes dynamically over time to effectively recreate hundreds or thousands of real-world conditions each equivalent to a single over-the-air test case.

Channel emulation-based performance testing is used extensively throughout all phases of product engineering. Examples of this type of testing include MIMO algorithm optimization, data performance (throughput vs. range, roaming), voice application performance (voice quality, dropped calls) and video application performance (video and audio quality). As all of these tests are conducted in varied multipath and fading environments, they provide an accurate means of characterizing real-world performance.

For 802.11n, the IEEE has recommended six specific models emulating typical indoor scenarios common to Wi-Fi installations. To demonstrate device performance in these different environments and the use of channel emulation to characterize device performance, a device manufacturer needs only to test the throughput vs. range performance of a MIMO device across all IEEE channel models. Figure 1 provides the results of a throughput vs. range test of an 802.11 draft n 2 × 3 MIMO device using the ACE 400NB channel emulator. The product was tested for all IEEE-defined channel models.

As illustrated in the plot of Figure 1, the maximum throughput of this device, which went from 75 Mbps to above 130 Mbps, varied greatly depending on the operating environment.

Critical requirements for channel emulation

A channel emulator must have dynamic emulation to mimic the constantly changing OTA channel conditions and a real-time path to precisely represent the inherent bidirectional nature of the device to access point path. Table 1 provides an “at-a-glance” look at the critical engineering and management requirements for channel emulation of 802.11n devices and the corresponding channel emulator or test system feature that addresses each requirement.

Functional testing

Functional testing in a static channel environment provides the necessary conditions for rapid development and troubleshooting of layers 2 and 3 of an 802.11n system. Testing devices in the simplified and stable RF conditions provided by a static channel tester like the Azimuth ADEPT-n MIMO functional test platform allows engineers to concentrate on features and functions, like basic association handshakes, security negotiations and roaming or hand-off, that are independent of layer 1 influences. Abnormalities in device behavior that are revealed through testing in a static channel environment can be easily attributed to the correct source without fear that the ever-changing conditions of a real-world dynamic channel environment could have affected the test results. Effective functional testing needs a controlled RF environment to provide repeatable test conditions that enables the comparison of tests run in different permutations, different test runs, different versions, different products or even sites under the same test conditions.

Functional MIMO testing involves four critical components:

  1. Protocol testing. Verifies that device operation conforms to standard specifications across communication states. This type of testing requires extensive packet capture, filter and 802.11n decode analysis capabilities.

  2. Interoperability testing. Verifies that APs and client devices designed by different vendors/manufacturers, supporting next-generation 802.11n as well as legacy 802.11a/b/g technologies, work well together.

  3. Mobility testing. Verifies operation of 802.11n client devices in mobile operating conditions. Examples of such tests include throughput vs. range and AP-to-AP roaming.

  4. Application-level feature testing. Verifies that the device delivers a quality user experience. Test examples would be basic data throughput and voice call quality measurement.

Testing in a controlled RF configuration (cabled) with static channel simulator (ideal conditions) can be used to measure the optimal performance of devices such as throughput, roaming and much more. As an example: In the case of roaming performance, the test will accurately measure the time that it takes a client to roam between APs. This provides a quick way to assess whether roam times fall within the guidelines established by the Wi-Fi Alliance VoWi-Fi certification or the IEEE 802.11r standard. Such a test may also look at roam efficiency and repeatability to understand the reliability of the roaming algorithm for real-time applications like VoIP.

Figure 2 provides the results of a walking test between two APs conducted on an 802.11 draft-n station using the ADEPT-n MIMO functional test platform and using the company's smooth roaming benchmark test.

The plot in Figure 2 shows the distribution of more than a thousand roams by a station between two APs. This is a normal distribution where ~50% of the roams occur in less than 35 ms and over 97% of the roams occur in less than 50 ms, a critical roam time specified in 802.11r.

Do-it-yourself vs. automated

There are two common methods to conducting MIMO functional testing. The first is a do-it-yourself (DIY) test approach that generally involves basic, low cost, over-the-air test setups that provide a quick way to start testing. While these solutions may initially be low cost, the value of the testing they complete is also low as testing in an uncontrolled RF environment produces inaccurate test results that cannot be repeated. More advanced DIY test solutions, using disparate hardware and software components including RF attenuators, switches, RF enclosures and packet capture, will perform testing in a controlled RF environment. However, the increased complexity of this type of test setup significantly raises installation and support costs. DIY test methods also typically rely on manual test execution and lack standard data management capabilities.

In the development of 802.11n MIMO systems, the large increase in operational and performance complexity has significantly augmented the number and complexity of functional tests required. With manual test execution, it is nearly impossible to achieve the thorough functional test coverage that is necessary to produce a high-quality 802.11n product using DIY test setups.

Standard test platforms, designed by wireless test specialists, have distinct advantages over alternative DIY approaches, as they are purpose-built to offer robust functional test capabilities at lower cost and lower complexity than DIY solutions. Effective standardized test platforms will integrate seamlessly with RF isolation chambers to ensure that accurate and repeatable test results are produced.

With the use of test automation, standard performance test scripts significantly reduce the time required to test 802.11n devices. Furthermore, with the increased functional and performance complexity of 802.11n devices, automated test is the only way to achieve the test coverage necessary to design high-quality 802.11n products.

Standard test platforms will often benefit from a data management tool that enables test data collection, sharing and correlation within an organization, as well as between partners. With the ability to actively engage internal engineering teams as well as external supply chain partners in product test, problem identification and debug, standardized automated test platforms significantly increase the efficiency of product engineering activities. Table 2 provides an “at-a-glance” comparison between MIMO test methods.

ABOUT THE AUTHOR

Graham Celine is a senior marketing director at Azimuth Systems Inc.

Table 1. Channel emulator requirements.
Critical Requirements for Effective and Efficient Channel Emulation Key Channel Emulator System Feature(s)
Accurate representation of over-the-air conditions Dynamic with very long intervals of statistically non-repeating data
Use standard channel models (IEEE models A-F for 802.11n) as well as custom, user-defined models
Bidirectional and reciprocal channel emulation
Test devices with multiple antennas 4×4 MIMO channel emulation
Precise RF impairment generation Channel emulator should introduce impairments like the real-world, but not interference (noise)
High fidelity RF - EVM better than the device under test, wide dynamic range and low noise floor
Simple troubleshooting Channel model play/forward/rewind/pause control
Test automation Integration with system level test automation and results database
Table 2. Comparing MIMO test methods.
Test Type Core Tests Channel Environment Test Efficiency (Coverage/Time)
Over-the-air Operational spot check Dynamic Very low
Performance testing MIMO algorithm optimization Dynamic High
Data performance (rate vs. range, roaming) in different environments
Voice, video application performance in different environments
RF design validation and QA
Functional testiing 802.11n layer 2 operation (MAC functionality) Static High
802.11n layer 3 operation (network operations, security, etc.)
Device/network interoperability
Mobility performance testing (roaming)
Application feature testing (throughput, voice quality)