SCT/ICT Configuration Tool Testing at UCAIug IEC 61850 IOP Events: Methodology, Evolution and Results

The main conclusion from the fifteen-year IOP history: the quality of SCL files — above all ICD files generated by vendor ICT tools — remains the critical bottleneck in multi-vendor integration. At IOP 2019 only 10 out of 159 ICD files (6.28%) passed validation without errors, leading to a fundamental reassessment of UCAIug's testing philosophy: since 2020 ICT functionality has become a mandatory part of IED certification testing. In parallel the role of machine-readable validation rules (OCL) has grown, and IOP 2024 identified more than 120 issues, the vast majority related to SCL files. These results are fed directly into IEC TC 57 WG10 through the User Feedback Task Force mechanism and the TISSUE database, influencing the development of the IEC 61850-6 standard itself.

History and Timeline of UCAIug IOP Events

UCA International Users Group (UCAIug) is a non-profit organisation promoting the integration and interoperability of power-system solutions based on international standards. The organisation serves as an umbrella for three user groups: CIM, IEC 61850 and OpenFMB. In the IEC 61850 domain UCAIug performs three key functions: accreditation of test laboratories for conformance testing, certification of conformance-test results, and organisation of Interoperability Testing (IOP) events.

IOP events have been held biennially since 2011. The fundamental difference between IOP and conformance testing is that conformance tests verify whether a specific device complies with the standard, whereas IOP aims to identify those areas of the standard that allow different interpretations and may prevent information exchange between certified devices from different vendors.

It is important to note that starting from the 2021–2022 cycle, each IOP is split into a virtual SCL component (testing configuration-file exchange) and an in-person F2F component (functional testing with real equipment), held several months apart.

When Configuration-Tool Testing Became a Separate Discipline

The evolution of the approach to SCT/ICT testing is one of the most telling trends in IOP history.

At IOP 2011 in Paris, SCL exchange testing was mandatory for all participants (every vendor had to provide an ICD and/or CID), but it was conducted as an ancillary activity without a dedicated track. The main conclusion: "There is no fully functional SCL validation tool" — XML validation proved insufficient for verifying compliance with IEC 61850-6, and implementations could not correctly interpret the XSD without a deep reading of the standard.

At IOP 2013 in Munich the engineering workflow (top-down and bottom-up) became one of the main test areas for the first time. It was here that "numerous problems related to device configuration and the SCL-based engineering process" (according to DNV) were discovered, acting as a catalyst for the development of formalised testing procedures for tools.

In 2015 DNV GL initiated the development of the first conformance procedures for SCT and ICT: "DNV GL is working with the UCAIug to develop the first release of the conformance test procedures for Substation engineering and device configuration tools." At IOP 2015 in Brussels, with ENTSO-E participation, the first in-depth testing of SCL processes was carried out, and the results were immediately presented at the IEC TC57 WG10 meeting, which had been strategically scheduled for the following week in the same ENTSO-E building.

A formally dedicated SCT track appeared at IOP 2019: the event was split into two independent test streams — (1) SCL exchange between System Configuration Tools and (2) integrated application. This was the turning point after which configuration-tool testing acquired a systematic character.

SCT/ICT Testing Methodology

Procedural Framework

SCT/ICT testing rests on several layers of documentation. The UCAIug Test Procedure Working Group (TPWG) develops and maintains test procedures, tracking issues through the Redmine system (redmine.ucaiug.org) with dedicated sub-projects: SCL Tooling, Server, Client, GOOSE Performance. The procedures for SCT and ICT were developed by DNV GL on the basis of Edition 2, accredited by UCAIug, and include test cases with indices such as sCnf (conformance), sDs (DataSet), sSvp (Sampled Values), etc.

The base standard IEC 61850-10 (Ed.2, 2012) defines methods and abstract test cases, but the specific procedures for SCT/ICT went beyond its scope and were developed by UCAIug together with manufacturers. As the standard has evolved, they have been supplemented by IEC 61850-6-3 rules — machine-readable SCL verification rules based on OCL (Object Constraint Language), which were first widely applied at IOP 2024.

SCL File Types in Testing

All six IEC 61850-6 file types participate in IOP testing, although their role has changed substantially from event to event:

Tested Engineering Workflows

Top-down approach — the primary method promoted by the standard and the one most actively tested at IOP since 2019. The process involves creating an SSD with a single-line diagram, importing ICD files from all vendors into the SCT, assigning IEDs to functions, configuring GOOSE/SV subscriptions, report blocks, communication parameters, and generating the SCD, then importing the SCD into vendor-specific ICTs to extract the CID, and finally loading the CID into the physical IEDs. At IOP 2021 this process was tested end-to-end with the involvement of the independent integrator POWER Engineers. One SCT worked top-down (creating virtual IEDs → defining data flows → overlaying real ICDs), the other bottom-up (from ICD files to signal binding).

Bottom-up approach involves individual IED configuration through vendor-specific ICTs, generation of IID/CID files, and subsequent import into the SCT for system integration (addressing, interconnections). This approach remains necessary when top-down tools cannot handle vendor-specific subscription features.

Round-trip engineering — iterative exchange between SCT and ICT, where system-level changes (SCD) are passed back to the ICT (via IID), and ICT changes are returned to the SCT. At IOP this process was tested implicitly with each SCD → ICT → feedback → SCD correction cycle. At IOP 2021 POWER Engineers described the process as follows: the SCD was handed to vendors at checkpoints, vendors imported it into their ICT, reported errors, and an iterative exchange cycle followed.

Evolution of Testing Across Standard Editions

The Edition 1 Era (IOP 2011)

IOP 2011 worked exclusively with Ed.1 SCL (2003 schema). Only the file types ICD, SSD, SCD and CID were available — without IID and SED. Core testing covered MMS Client/Server, basic GOOSE, and initial SV (9-2). SCT/ICT testing was limited to CID-file exchange and basic SCD creation. Key Ed.1-era findings: XSD validation is not equivalent to SCL validation; there was no validated XSD release with TISSUE corrections; "a good percentage of discovered problems had already been resolved in Edition 2."

Transition to Edition 2 (IOP 2013, 2015, 2017)

IOP 2013 was the first event at which Edition 2 capabilities were tested: new file types IID and SED, backward compatibility with Ed.1, and an extended Services section. Issues recorded in Redmine and forwarded to WG10 included: namespace handling (Issue #445), mixing Ed.1 and Ed.2 data models in a single SCL file (Issue #444), unclear ClientLN format for RCB reservation (Issue #446), use of ExtRef for GOOSE and SV subscription (Issue #445), and data-type management in DataSets for GOOSE subscription (Issue #463).

At IOP 2015, with ENTSO-E participation, a generalised engineering process was developed and test cases derived from it — for the first time the SCL area was tested so thoroughly. Results were immediately conveyed to WG10: "most problems discovered at IOP 2015 were resolved during the WG meeting." Moreover, the IOP results became one of the reasons for delaying the publication of IEC 61850-6 Ed.2.1 CDV — in order to incorporate the fixes in time.

IOP 2017 in New Orleans brought a fundamental paradigm shift: the move from pair testing to an integrated application — modelling a real substation with devices assigned to functions. SCT tools were used to design the multi-vendor system, although a formal SCT test plan had not been prepared (this was noted as a lesson learned for IOP 2019). Among the issues discovered was the absence in IEC 61850-6 of a way to describe the authentication configuration for a client (Issue IOP_2017/8).

The Edition 2.1 Era (IOP 2019, 2021–2022, 2023–2024)

IOP 2019 was a turning point for SCT/ICT testing. The result of validating 159 ICD files with 5 tools — only 6.28% without errors — led to a fundamental reassessment of testing philosophy: ICT functionality was included in mandatory IED conformance tests for Ed.2 Amendment 1. Separate conformance tests for SCT and ICT were also strengthened.

At IOP 2021–2022 two SCTs worked in parallel for the first time to create an SCD for the same substation. One SCT used the top-down approach (virtual IEDs → data flows → overlay real ICDs), the other bottom-up (from ICDs to signal binding). The independent integrator POWER Engineers served for the first time as a neutral SCD developer. For functional testing in Milan 7 end-to-end digital-substation tests were conducted, including a custom test by the Belgian TSO Elia, which revealed an unexpected issue: some IEDs blocked protection upon loss of PTP synchronisation. First-time tests included: a functional ADMS based on IEC 61850, and secure R-GOOSE over a routed environment with a Key Distribution Center (KDC).

IOP 2023–2024 marked several major innovations. The virtual SCL IOP (December 2023) was the first to make wide use of OCL rules for ICD-file validation, and also tested automatic network-equipment configuration based on the SCD file. The in-person IOP in Birmingham (September 2024) introduced the Basic Application Profiles (BAP) approach per IEC 61850-7-6: instead of one large substation, typical bays were designed with IED roles (LPU, BCU, PIU) and arbitrary vendor combinations. The SSD file specified all functions (LNs) and signals; the BAPs defined which signals a device with a given function should produce or consume. However, as Christoph Brunner (WG10 convenor) noted: "During the preparation of IOP24 we found that many ICD files from various vendors contained incomplete or incorrect Services elements" — in particular, problems with ClientServices (when the goose attribute is absent, the SCT considers the device incapable of subscribing to GOOSE) and valKind/valImport (devices still use valKind="Conf", which Brunner considers unacceptable for modern IEDs). Of the 120+ issues found, the majority related to SCL and were detected by the new OCL rules.

Results, Issues and Participating Tools

SCT and ICT That Participated

It should be emphasised that complete participant lists are not published — the data presented here are based on publicly available sources (PAC World articles, vendor blogs, LinkedIn).

Typical Tool-Interoperability Issues

Namespace handling — a chronically problematic area. Vendors use proprietary namespaces in private sections without providing the corresponding XSD schemas, making correct validation impossible. IEC 61850-7-1 (Section 14.2) defines namespaces for ldNs, lnNs, dataNs, cdcNs — vendor-specific namespaces may only be modified by the vendor's ICT, not by the SCT. At IOP 2013 mixing Ed.1 and Ed.2 data models in a single SCL file (Issue #444) was one of the central issues.

Private sections remain a source of incompatibility. SCT tools often cannot process proprietary data in Private elements from IED vendors. When protection settings are modelled in private sections (rather than through standard IEC 61850 data objects), the SCT cannot manipulate or preserve this data during file transformations. Information loss during SCL-file conversion (ICD → SCD → CID) is a documented problem: a number of tools lose descriptions (desc), units of measurement (unit), device names (IEDName) or multicast-subscriber declarations.

DataTypeTemplates mismatches occur when manufacturers modify or extend the standard's data types without following established procedures. Non-standard data types are not recognised by equipment from other vendors. A typical validation error: "FCDA does not refer any existing DA or BDA in DataTypeTemplates section." Some manufacturers modify the XSD to relax constraints — files pass internal validation but are rejected by other tools.

Address and control-block conflicts: GOOSE AppID, MAC addresses and IP addresses must be coordinated through the SCT, but different vendors implement GSESettings, ReportSettings and SMVSettings differently (Fix vs. Conf vs. Dyn). Violation of naming conventions (length, special characters) is a common problem that not all tools detect.

GOOSE/SV subscription configuration is the "most complex part" of multi-vendor testing (GE Vernova). Subscription binding through ExtRef elements differs significantly between vendors. In practice a bottom-up approach is often required, where each ICT independently configures the reception of GOOSE messages from other vendors' devices.

Handling large SCD files: real-world SCDs can contain more than one million lines of code, making them virtually uninterpretable by humans. On large projects (e.g. 2,300 IEDs) performance problems and SCT crashes have been reported, in some cases requiring the project to be split into parts.

The Services section — an issue that resurfaced acutely at IOP 2024. Many ICD files contained incomplete or incorrect Services elements, preventing SCTs from determining the permitted operations with a device. The situation with ClientServices is particularly critical: when the goose attribute is absent (default = false), the SCT considers the device incapable of subscribing to GOOSE. valKind/valImport issues also persist: devices still use valKind="Conf" (value in SCL but not visible in the communication data model), which is incompatible with the expectations of modern SCTs.

Achievements and Positive Trends

Despite the chronic problems, the testing dynamics show steady progress. According to IOP 2015 data, only 4 issues were identified in the Client/Server area (vs. 15 in 2013); no GOOSE issues were found at all. ENTSO-E stated: "IOP showed that parts of the IEC 61850 standard and products are very stable … there is no need to wait to deploy IEC 61850."

The involvement of an independent integrator (POWER Engineers since 2021) validated the engineering workflow from a user rather than a vendor perspective. The emergence of open-source SCTs (OpenSCD/CoMPAS, an LF Energy project) represents a paradigm shift: Transpower NZ by 2024 had implemented a complete workflow based on OpenSCD, creating a single SCD file for the entire substation without vendor lock-in.

Automatic network-equipment configuration from the SCD file, first demonstrated at IOP 2024, opens the way to significant reductions in digital-substation commissioning costs. The first multi-vendor secure R-GOOSE with key and security-policy distribution is another important achievement of IOP 2024.

Impact on the IEC 61850 Standard and Related Documents

The relationship between IOP and the development of the standard is bidirectional and close. The direct channel (IOP → WG10) is implemented through the IEC 61850 User Feedback Task Force, maintained by UCAIug via the Redmine system, and through the TISSUE database (Technical ISSUE, iec61850.tissue-db.com). Standard issues discovered at IOP are formally submitted through national IEC committees to TC57/WG10. Example: all 9 standard issues from IOP 2019 were submitted, reviewed and resolved by WG10. At IOP 2015 results were presented to WG10 the day after testing ended, and "most problems were resolved during the working-group meeting." The delay in publishing IEC 61850-6 Ed.2.1 CDV is partly explained by the need to incorporate solutions found at IOP.

The reverse channel (WG10 → IOP) is expressed in the fact that WG10 develops OCL validation rules that are tested at IOP, and the test results determine the need for additional rules. IEC 61850-6-3 (under development) formalises these machine-readable rules. Personnel overlap strengthens the integration: Herbert Falk is simultaneously UCAIug Vice-President for Testing and an IEC 61850 editor; Christoph Brunner is WG10 convenor and an active IOP participant/analyst.

IOP has also influenced the development of IEC 61850-6 Amendment 2 (2024), which introduced UUID identification of elements (ICT creates a UUID for elements in ICD; SCT generates a new instance UUID in the SCD context while preserving the template UUID), the SclFileReference element for cross-references between SCL files, and improved engineering-rights tracking.

IOP results are reflected in CIGRE technical brochures: TB 819 (WG B5.50, 2020) — user expectations and stakeholder interaction for IEC 61850-based automation systems, noting that "due to limited interoperability of IEC 61850 products and high standard complexity … a number of projects encountered serious difficulties." WG B5.68 (Terms of Reference, 2018) — "Optimisation of the IEC 61850 PACS Engineering Process and Tools" — directly addresses configuration-tool interoperability requirements, engineering-process harmonisation, the boundary between harmonisation and customisation, and use cases for partial configuration modification. TB 401 provides a structured method for functional testing of IEC 61850 systems.

Timeline of SCT/ICT Testing Evolution

Conclusion: Strategic Insights for the Configuration-Tool Market

The fifteen-year IOP history reveals several fundamental trends critically important for SCT/ICT developers. First, ICD/IID file quality is the Achilles' heel of multi-vendor integration, and the inclusion of ICT testing in mandatory IED certification creates a new market imperative for IED vendors. Second, machine-readable validation (OCL) is displacing XSD validation as the primary tool for SCL-file checking — investment in support for IEC 61850-6-3 OCL rules is becoming strategic. Third, the BAP approach (IEC 61850-7-6) and automatic network-equipment configuration from SCD define the development trajectory for digital-substation engineering over the coming years. Fourth, the emergence of open-source SCTs (OpenSCD) and independent integrators signals market deconcentration in configuration tools, where competitive advantage lies not only in functionality but also in the ability to achieve seamless multi-vendor integration.

Finally, every IOP confirms: top-down engineering remains the right approach, but practical obstacles to its full implementation — differing interpretations of the standard, issues with private sections and Services elements — still require hybrid solutions. IOP 2026 in Europe, given the further development of IEC 61850-6-3 (OCL), IEC 61850-90-30 (functional modelling) and Edition 2 of IEC 61850-7-6 (BAP in SCL format), promises to be the next maturity checkpoint for configuration tools.