To meet the network requirements of 5G and beyond, virtual network functions (VNFs) are replacing physical network functions. Yet orchestrating VNF’s is a complex management process. To accomplish this, network function virtualization management and orchestration (NFV MANO) solutions are being developed.
MANO systems affect the overall performance of NFV systems. To manage their complexity, the ETSI Industry Standard Group for VNFs (ESTI ISG NFV) has introduced a standard MANO system framework as a reference for vendors and those working with Open Source MANO (OSM) projects.
Unlike traditional networking functions and systems, a NFV MANO system has no well-defined test methods or key performance indicators (KPIs) to benchmark, measure and validate its success. To resolve this, NEC Laboratories Europe has introduced formal NFV MANO benchmarking methodology and key performance indicators (KPIs). To test our proposed KPIs and methodology, we compared and analyzed the performance of two popular Open Source NFV MANO projects: 1) Open Network and Automation Platform (ONAP) and 2) Open Source MANO (OSM) using open source, virtual customer premises equipment (vCPE) as a common use case.
Challenges of Benchmarking NFV MANO Systems
A traditional, network management system (NMS) uses FCAPS management to monitor, measure and enforce the network and service KPIs that they are managing. These include network and service availability, service level agreements (SLAs), latency, jitter and errors. KPIs also measure the performance of network function virtualization infrastructure (NFVI) and VNFs such as compute resources, network resources and virtualized router functions. These are well known and well defined. NFV MANO systems also perform Life Cycle Management (LCM) of network services (NS) and VNFs. However, the lack of well-defined KPIs needed to correctly benchmark the efficiency of NS and VNF management operations poses a challenge.
Virtualized services, delivered by VNFs and network services, are highly agile and dynamic. When monitoring events, delivering and executing appropriate LCM actions will depend on the reaction time of a NFV MANO system. Quick reaction time is essential and ensures that customers can achieve their performance requirements. This includes customers from different verticals who are sharing the same NFVI resources.
Benchmarked correctly, MANO systems enable customers to make informed decisions when choosing an appropriate MANO solution for their operational needs.
MANO Performance KPIs
Establishing MANO benchmarks and KPI’s are an important but complex undertaking. NLE has proposed classifying MANO system key performance indicators into two categories: Functional KPIs and Operational KPIs.
Functional KPIs describe non-runtime characteristics of a MANO system and include:
- A resource footprint of the deployed MANO system
- The variety of virtualized infrastructure manager (VIM) platforms that a MANO system can support
- The number of VIMs a single MANO platform can manage efficiently
- The maximum number of VNFs a MANO system can effectively and efficiently monitor and manage in an NFVI
- Available features for example, support for DevOps, VNF package management, VNF image management and integrated monitoring systems
Figure 1. KPI’s for a NFV MANO system
Operational KPIs characterize run-time operations. These are quantified by measuring the time delay of a particular LCM action or task and its effectiveness. Operational KPIs include:
- On-boarding process delay (OBD)
- Deployment process delay (DPD)
- Run-time orchestration delay (ROD)
The definition of operational KPIs are:
On-boarding process delay (OPD)
The time it takes to boot-up a virtualized network function image such as a virtual machine with all its resources.
Deployment process delay (DPD)
The time it takes to deploy and instantiate a VNF within the booted virtual machine and setup an operational network service.
Run-time orchestration delay (ROD)
Run-time orchestration operations consist of different management actions. The performance delay of each individual action can be quantified by measuring the time difference from the moment the action is executed to the time the action is completed (see Figure 2).
NFV MANO Benchmarking Environment
In order to test the proposed KPIs, NEC Laboratories Europe has benchmarked and compared the performance of two popular Open Source MANO projects, ONAP-Beijing release (ONAP-B) and OSM Release 4 (OSM-4). Figure 2 shows the laboratory test environment for these scenarios. To ensure a fair and comparative analysis we used a common VIM and a common VNF for both ONAP-B and OSM-4 test beds. For the VIM, we used cloud operating system, OpenStack Ocata, while for the VNF we used virtual Customer Premises Equipment (vCPE).
Figure 2. NEC Laboratories Europe Laboratory Setup Overview
Figure 3. Deployment process delay for vCPE, VNFS
In Figure 3, we compare the deployment process delay of the individual vCPE, VNFCs for OSM-4 and ONAP-B. Although an aggregate, the overall difference is not significant. For VNFCs, there are some measurements where the deployment process delay (DPD) is higher in OSM-4 than ONAP-B and vice versa. VNFCs with services and network requirements, such as vINFRA and vBNG, have generally a higher DPD when compared to the other vCPE, VNFCs. A detailed discussion about this and other proposed KPI results and benchmarking methods are described in the research paper, “Benchmarking open source NFV MANO systems: OSM and ONAP (G. Yilma et al.).”
Outlined in Table 1, “OSM-4 versus ONAP-B Comparison Matrix” are the lessons learned from our comparison between OSM-4 and ONAP-B. When comparing these two systems, we observe that OSM-4 has a much lower resource footprint. While the learning curve is higher for ONAP-B, it has a much richer feature set which translates into a broader set of functionality than OSM project. For example, the Service Design Center (SDC) has a robust closed-loop monitoring system to generate polices. It can also integrate other advanced artificial intelligence applications.
Table 1. OSM-4 versus ONAP-B Comparison Matrix
In this blog, I have discussed the importance of analyzing the performance of NFV MANO systems and highlighted the challenges that this analysis entails. We can maximize the performance of NFV MANO systems by specifying a set of functional and operational KPIs that benchmark their performance. NLE tested the validity of this KPI framework by defining the KPIs of two popular and competing open-source NFV MANO systems, OSM-4 and ONAP-B and analyzing their performance. To ensure a fair comparison, a common OpenStack VIM platform was used that utilized the same type of vCPE, VNF.
NLE’s evaluation provided valuable performance insights about OSM-4 and ONAP-B, and highlighted large feature discrepancies between what was claimed by each system and what actually worked. According to our evaluation, both platforms can successfully on-board and instantiate network services and VNF instances.
The operational and functional performance of OSM-4 and ONAP-B were compared. Operational KPI’s analyzed the resource footprint of OSM-4 and ONAP-B by measuring their on-boarding process delay (OPD) and deployment process delay (DPD). We also evaluated the functional performance of OSM-4 and ONAP-B by comparing the types of VIMs (e.g., OpenStack, AWS, etc.) that the two systems can support and the number of VNF’s that they can manage and orchestrate.
Run-time orchestration actions such as scaling operations failed on both OSM-4 and ONAP-B systems. As such, KPI performance could not be analyzed for run-time orchestration delay (ROD) and Quality of Decision (QoD). We expect run-time orchestration functionalities to be supported in future releases. Further details can be read in the research paper “Benchmarking open source NFV MANO systems: OSM and ONAP (G. Yilma et al.)” available here.