Javascript is required
1.
F. L. Alemnge, L. C. Forje, N. A. Symphorien, B. Liudmyla, A. Avny, J. E. J.-E. Lane, A. Dan, C. Gomes, Z. Sohrabi, N. Zarghi, M. Lesage, G. Raîche, M. Riopel, F. Fortin, D. Sebkhi, and Y. Otis, New Horizons in Education and Social Studies. Book Publisher International, 2020. [Online]. Available: [Google Scholar] [Crossref]
2.
S. Ghosh, S. Thomke, and H. Pourkhalkhali, “The effects of hierarchy on learning and performance in business experimentation,” Acad. Manag. Proc., vol. 2020, no. 1, p. 20500, 2020. [Google Scholar] [Crossref]
3.
M. Lesage, “Hierarchical Aggregate Assessment (HAA): An assessment process of teams with several levels of hierarchy in education,” Creat. Educ., vol. 07, no. 14, pp. 1974–1994, 2016. [Google Scholar] [Crossref]
4.
G. Sammour, A. Al-Zoubi, A. Gladun, K. Khala, and J. Schreurs, “Semantic Web and ontologies for personalisation of learning in MOOCs,” in 2015 IEEE Seventh International Conference on Intelligent Computing and Information Systems (ICICIS) December 12-14, 2015, Cairo, Egypt, 2015, pp. 12–14. [Google Scholar] [Crossref]
5.
W. D. Nance, “Improving information systems students’ teamwork and project management capabilities: Experiences from an innovative classroom,” Inf. Technol. Manag., vol. 1, no. 4, pp. 293–306, 2000. [Google Scholar] [Crossref]
6.
H. Eysenck, Personality Theory and the Problem of Criminality. 1987. [Google Scholar]
7.
J. R. Graham, The MMPI: A Practical Guide. Oxford University Press, 1987. [Google Scholar]
8.
“Development of theoretical foundations for the modern computerized organization of the labor team,” 2021. https://www.sworld.com.ua/simpua17/sua17-1.pdf [Google Scholar]
9.
A. Y. Gladun and J. V. Rogushina, “Use of ontological analysis for evaluation of expert competences in the domain of national standards develoment,” Syst. Res. Inf. Technol., no. 3, pp. 19–32, 2016. [Google Scholar] [Crossref]
10.
O. V. Strokan, S. M. Pryima, J. V. Rogushina, A. Y. Gladun, D. V. Lubko, and A. A. Mozgovenko, “AdvisOnt: Semantization of Agricultural Advisory Services for Validation of Outcomes of Non-Formal and Informal Learning. I,” Control Syst. Comput., no. 1 (291), pp. 62–70, 2021. [Google Scholar] [Crossref]
11.
A. Hladun, Y. Rohushyna, and I. Subach, “Anontology modelling human resources management for innovational domains,” Collect. ”Inf. Technol. Secur.”, vol. 6, no. 1, pp. 15–25, 2018. [Google Scholar] [Crossref]
12.
A. Gladun, K. Khala, and R. Martinez-Bejar, “Development of object’s structured information field with specific properties for its semantic model building,” CEUR Workshop Proc. (ceur-ws.org), vol. 3241. pp. 102–111, 2022. [Google Scholar]
13.
M. Lesage, G. Raiche, M. Riopel, F. Fortin, and D. Sebkhi, The Internet Implementation of the Hierarchical Aggregate Assessment Process with the “Cluster” Wi-Fi E-Learning and EAssessment Application — A Particular Case of Teamwork Assessment. InTech, 2015. [Online]. Available: [Google Scholar] [Crossref]
14.
K. C. Laudon and J. P. Laudon, Management Information Systems: Organization and Technology. Prentice-Hall, Inc., 1995. [Google Scholar]
15.
K. Willey and M. Freeman, “Improving teamwork and engagement: The case for self and peer assessment,” Aust. J. Eng. Educ., vol. 12, no. 2, pp. 1–19, 2006. [Google Scholar]
16.
K. C Laudon, J. Laudon, and M. Brabston, Management Information Systems: Managing the Digital Frm. Pearson Canada, 2011. [Google Scholar]
17.
H. Van Zyl and L. Massyn, “Integrated assessment: A learning adventure and growth opportunity for adult learners,” Am. J. Bus. Educ., vol. 1, no. 2, pp. 95–104, 2008. [Google Scholar] [Crossref]
18.
R. W. Lingard, “Teaching and assessing teamwork skills in engineering and computer science,” J. Syst. Cybern. Inform., vol. 8, no. 1, p. 35, 2010. [Google Scholar]
19.
D. Gijbels, F. Dochy, P. Van den Bossche, and M. Segers, “Effects of problem-based learning: A meta-analysis from the angle of assessment,” Rev. Educ. Res., vol. 75, no. 1, pp. 27–61, 2005. [Google Scholar] [Crossref]
20.
A. Gladun and J. Rogushina, “Data mining – finding knowledge in data,” 2016. https://www.researchgate.net/publication/304025285_Data_Mining_Search_for_Knowledge_in_Data [Google Scholar]
21.
D. V. Lande, I. Y. Subach, and Y. E. Boyarynova, “Fundamentals of the theory and practice of intelligent data analysis in the field of cybersecurity,” 2018. https://ela.kpi.ua/bitstream/123456789/45721/1/NPOsnovy_teori_i_intelekt_analizu.pdf [Google Scholar]
22.
D. V. Lande and I. Y. Subach, “Visualization and analysis of network structures,” 2020. http://dwl.kiev.ua/art/vams/vams.pdf [Google Scholar]
23.
A. Gladun, J. Rogushyna, and M. Lesage, “A method for assessing the effectiveness of cybersecurity information and analytical support systems,” Coll. ”Inf. Technol. Secur., vol. 10, no. 2, pp. 1–13, 2022. [Google Scholar] [Crossref]
24.
J. Rogushina and S. Priyma, “Use of competence ontological model for matching of qualifications,” Chem. Bulg. J. Sci. Educ., vol. 26, no. 2, pp. 216–228, 2017. [Google Scholar]
25.
“The European multilingual classifier of skills, competences, qualifications and qccupations,” 2022. https://ec.europa.eu/esco/portal/home [Google Scholar]
26.
A. Gladun, K. Khala, and R. Martinez-Bejar, “Development of object’s structured information field with specific properties for its semantic model building,” CEUR Workshop Proceedings (ceur-ws.org), vol. 3241. pp. 102–111, 2021. [Google Scholar]
27.
A. Gladun and K. Khala, “Ontology-based semantic similarity to metadata analysis in the information security domain,” Probl. Program., no. 2, pp. 034–041, 2021. [Google Scholar] [Crossref]
Search
Open Access
Research article

Hierarchical Aggregate Assessment of Multi-Level Teams Using Competency Ontologies

anatoly gladun1*,
julia rogushina2,
martin lesage3
1
International Research and Training Center for Information Technologies and Systems under NAS and MES of Ukraine, 03680 Kyiv, Ukraine
2
Institute of Software Systems, National Academy of Sciences of Ukraine, 03680 Kyiv, Ukraine
3
Faculty of Educational Sciences, Université du Québec à Montréal, H2X 3R9 Montréal, Canada
Acadlore Transactions on AI and Machine Learning
|
Volume 2, Issue 2, 2023
|
Pages 55-65
Received: 03-27-2023,
Revised: 05-09-2023,
Accepted: 05-23-2023,
Available online: 06-05-2023
View Full Article|Download PDF

Abstract:

It is complex to assess multi-level hierarchical teams, because the solution needs to organize their rapid dynamic adaptation to perform operational tasks, and train team members without sufficient competencies, skills and experience. Assessment also reveals the strengths and weaknesses of the whole team and each team member, which provides opportunities for their further growth in the future. Assessment of the work of teams needs external knowledge and processing methods. Therefore, this study proposed to use ontological approach to improve the assessment of multi-level hierarchical teams, because ontology integrated domain knowledge with relevant competencies of positions and levels in the hierarchical teams. Information on competencies of applicants was acquired in the portfolio analysis. After subdividing the hierarchical teams, appropriate ontologies and Web-services were used to obtain assessment results and competence improvement recommendations for the teams at various sublevels. The step-by-step team assessment method was described, which used elements of semantic similarity between different information objects to match applicants and equipment with team positions. This method could be used as a component of integrated multi-criteria decision-making and was targeted at specific cases of user tasks. The set of assessment criteria was pre-determined by tasks, and built based on domain knowledge. However, particular criterion were dynamic, and changed along with environmental at different time points.

Keywords: Ontological analysis, Competency matching hierarchical team assessment, E-learning, Semantic similarity

1. Introduction

It is a complex scientific issue to assess multi-level hierarchical teams, which are composed of individuals with different roles and competencies, because a wide range of criteria need to be used and various aspects of team work need to be considered. These teams include technical project teams, scientific research teams, military units, medical teams, educational institutions. Many of them have complex structure, whose levels and hierarchy are caused by domain specifics, various rules and standards, and goals of team activities. Therefore, external knowledge of this structure is needed, and then is turned into methods to evaluate the teams.

This study aimed to make the assessment methods of multi-level hierarchical teams more effective, using formalized semantic models of knowledge of the subject area (ontologies, taxonomies, thesauri), thus performing complex tasks, selecting relevant personnel and providing operational training to enable personnel to obtain certain competencies using the recommended courses. This study focused on developing a method to compare complex information objects, based on the semantic similarity method by taking into account multi-criteria selection.

2. Literature Review

Many organizations and teams have a hierarchical structure with several management levels. It is necessary to develop methods to assess those teams in an academic or work context, thus analyzing their potential possibility of collectively fulfilling the assigned tasks, and creating new project teams, military units, and expert commissions. Many researchers have assessed multi-level hierarchical organizations using mathematical calculation [1]. However, this issue requires a new approach, i.e., applying the knowledge of subject semantic modeling obtained from ontological analysis when creating information systems.

The analysis of Ghosh et al. [2] showed that the assessment approach of organizations, collectivities and work teams were used in many human activity fields, particularly formation of military units, resuscitation medical teams and research teams, as well as training of doctors and teachers [3]. All individuals were assessed in the same way in the traditional assessment paradigm, such as universities, formal and informal education [4]. However, hierarchical cumulative (aggregate) assessment varied depending on the hierarchical level of team members. This type of assessment included two aspects: the performance of assigned tasks, and accumulated experience, skills and abilities concerning team management (e.g., leadership skills, ability of organizing tasks, achieving goals and obtaining positive work results), which required changes to the generally accepted assessment theories and paradigms [5]. Thus, this study presented the basics of Hierarchical Aggregate Assessment (HAA) paradigm.

3. Theoretical Framework

The problem of creating and evaluating of organizations has many aspects, which should reflect both the characteristics of individual personalities and the relations between them, including grouping relationships that allow creating multi-level hierarchical structures. For this purpose, the results from such areas as the study of the mechanisms of the structural and functional integrity of hierarchical regulation systems, adaptive response, optimization of human-machine systems, typological and biorhythmological dynamic portrait of the personality, etc. can be applied.

One of the important aspects is the understanding and formalization of the properties of an individual, his/her type and characteristic features. For this, two levels of knowledge can be applied: empirical information about human nature and theoretical-experimental knowledge about neurophysiological mechanisms that use objective indicators of conditioned reflex activity.

All personality theories are based on structural concepts that are used as the building blocks for modelling of .stable characteristics of the human psyche. One example is the concept of personality traits, that are considered as a stable qualities or tendencies of a person to behave in a certain way in different situations. The structure of personality can be described using the concept of personality type that can be defined as a set of traits that creates an independent category with clearly defined boundaries. Each human trait can be represented by a continuum of states or personality properties between two opposite values (for example, the pairs "will-determinism" and "introvert-extrovert" describe such continua). The type of personality can be determined by different sets of individual properties (for example, Eysenck [6] separates 3 properties, the MMPI test [7] takes into account 13 properties, Jung – 8 ones, etc.). Individual features are based on such fundamental properties of nervous processes of excitation and inhibition as their strength, balance and mobility. For example, the use of two continuums such as "introversion-extraversion" and "neuroticism-stability" allows us to distinguish four categories of people according to the types of higher nervous activity: sanguine - a strong, balanced, mobile type of nervous system; choleric - strong, unbalanced, mobile; phlegmatic - strong, balanced, inert; melancholic - weak, unbalanced, inert.

The study of the structural and functional organization of the team, which is aimed at assessing its adaptation to various factors of the external environment, is a more difficult task compared to the assessment of the individual adaptation of the team members [8]. It should ensure the individualization of information flows and means of interaction on the basis of knowledge of individual sociopsychophysiological (SPF) properties and functional states of team members on the basis of individual and typological adaptation. SPF factor is a dynamic set of sociological, psychological, and physiological properties inherent in an individual or team during the performance of a certain activity.

Evaluation of the organization and functioning of collectives based on knowledge about the structure of the personalities, their functional states and individual information spaces allows to determine the actual functional state of the collective and the level of optimal performance.

Research on the SPF factor is based on the theory of the individual-typological approach, as an innovative foundation of the theory of personality, and the biosocial culture of a person, which includes knowledge about the type of personality, its dynamic structure and individual adaptive information space, etc. This provides an objective assessment of the activity and training of team members, taking into account the balance of goals of the process participants with the choice of the level of detail of information from the chosen position.

On the basis of these properties, it is possible to create an SPF profile of both an individual and a team. But for large teams with many levels of hierarchy this task becomes more complicated and must be divided into a set of simpler subtasks, where the profile of one individual information object (IO) and its interaction with the IO of the same level, one level higher (direct management) and on one level below (direct subordinates). More complex situations are considered as complex IOs and analyzed based on the comparison of such IOs.

Work or performance of team members with hierarchical positions was assessed based on the design of assessment tasks for each level of hierarchy. Process of grouping team members into several levels of hierarchy and giving them hierarchical positions was called the aggregation process, which defined the concept of HAA. A special feature of the assessment was that all the different assessments were given simultaneously in one task with assessment sessions. E-learning courses and Massive Open Online Courses (MOOCs) [9] were used to assess teams as well as formal and informal training based on competencies [10].

In addition to assessing the knowledge of team members, special attention was paid to the assessment methods of organizational ability, i.e., the ability to manage a team, such as leadership skills, and the ability of organizing tasks. According to the assessment results of team members, it was possible to change the hierarchical structure of the team by appointing other applicants to leadership/management positions. To assess leadership skills, it was necessary to prepare simulation scenarios to assess managers in the team, and produce electronic portfolios of managers and other information on applicants.

HAA was also considered as a large electronic portfolio of the work produced by an organization or different teams in the organization, which represented the achievements or the performance of team members, productivity of employees (executives), performance degree of assigned tasks, points, assessments, self-assessment and peer review of team members.

Studies and open publications in this field are still quite few today. However, many researchers believed that the operation of an intelligent information system (IIS) was improved by a module, which ensured the use of knowledge of this field, provided coordination and assessments, and produced logical deductions based on knowledge [11]. A hierarchical system of ontologies was just such a module as a component part of IIS [12]. An ontology provided the management process of HAA, competency recommendations for each position or hierarchical level of the team, and services for the applicants to obtain necessary competencies by choosing courses. In addition, the assessment of knowledge and skills included grades, scores, performance, task completion, portfolio, as well as formal, formative and summative assessment results. The assessment of leadership skills may include different indicators specific to each organization's policies, such as hours worked, turnover, as well as number of produced and sold products and items in stock [13].

For large multi-level hierarchical organizations, they were assessed in teamwork by completing complex assessment tasks in collaborative mode. In particular, the assessment of large organizations in the field of computer programming involved a large amount of work [14]. Information systems of the organizations contained information on functions, departments and hierarchical positions of members of the organization, which was stored in hierarchical databases [15]. Employee files were sorted by their titles or assignments to identify those sharing the same hierarchical position. The structure of the database, which illustrates the positions of employees, is shown in Figure 1.

Figure 1. Records of an employee in a management information system database [16]
Note: This figure was prepared by the authors.

4. Methodology

The results of lots of papers are based on methods of artificial intelligence, computer science, system analysis, programming and algorithm theories, rather than the normal deduced research conclusions in humanities based on the experimentation on research subjects and the qualitative or quantitative analysis of results. However, the methodology used in this study was part of theoretical research inductions based on ontologies and computer algorithms. The results presented in the next sections were the ontological analysis produced in the HAA paradigm and its implementation in different kinds of ontologies. The analysis also led to a discussion and comparison with the standard assessment methods. Finally, this study ended with a conclusion and further work.

5. Results and New Deduced Theoretical Assumptions

The HAA concept was based on the aggregation process, course curriculum, portfolio assessment, assessment methods, leadership soft skills and linkage to management information systems, which aimed to assess teams with several levels of hierarchy. Information Artifact Ontology (IAO) was a special case of ontology, and the knowledge in IAO was used to support various forms of learning and assessment in the field of education.

5.1 General Aspect

This three-step assessment process was performed using a tree-like structure of the organization. The first step was the aggregation process, where a team was formed, and hierarchical positions were assigned to team members. The second step was providing a performance, test or assessment task to be completed by the team members. The third step was the overall team performance assessment with possible change or modification of roles or hierarchical positions of the team members, and then returning to the initial assessment step. The process was iterative and assessed competencies of the team members. If some team members did not obtain the competencies, training courses, formal and informal learning methods were proposed to them to achieve these competencies. Figure 2 shows the assessment process.

Figure 2. Steps of the assessment process and hierarchical position assignation
Note: This figure was prepared by the authors.

HAA was also performed online through E-learning or E-assessment applications. Teamwork assessment involved collaborative work and problem-based online learning. Collaborative learning [17], collaborative assessment work [18] and problem-oriented learning [19] were implemented by completing complex assessment tasks with several levels of hierarchy. Summative assessment included all marks and grades given by an assessor, who was usually the course teacher, professor, instructor or supervisor. The HAA process united team members into a team with several levels of hierarchy, where specialists occupied the hierarchical positions of manager (administrator or group leader), team leader and member. The team structure had the shape of a pyramid or an inverted tree, which was an organizational chart, with aggregation of team members as each branch.

To carry out the clustering process by calculating proximity indicators of characteristics of team members, the task of hierarchical clustering was performed with visualized results in the form of dendrograms [20]. Cluster analysis was based on the unique facts of observed values, while discriminant analysis [21] was a set of statistical analysis methods for pattern recognition (classification), which was used to decide which variables distributed (i.e., “discriminate”) the obtained data sets (so called "groups"). Unlike cluster analysis, the groups in discriminant analysis were known a priori. Cluster analysis had two methods: supervised and unsupervised clustering. The former ensured obtaining certain adapted classification, while the latter did not use classification [22].

5.2 Semantization of HAA

Semantization of IAO was proposed based on the knowledge of relevant ontologies. Therefore, it was planned to use the following ontologies in Table 1:

Table 1. Ontologies as a source of semantics for HAA

Ontologies

Knowledge obtained from ontologies

PRO

Main concepts of Publishing Roles (PR), their properties and the relationship between them

Organizational ontologies

Hierarchical structure of an organization, its departments, divisions, staff and the relations (including hierarchical) between the organization’s members

Competency ontologies

Essential skills, experience and professional abilities corresponding to the selected domain

- Publishing Roles Ontology (PRO) included the operating mode of the evaluated team;

- Organizational ontologies reflected the specific characteristics of certain types of teams (e.g., training classes, work groups, military units, research teams), their hierarchy and the semantics of relationships between team members;

- Competency ontologies, such as European Skills, Competences, Qualifications and Occupations (ESCO), formalized information on education, experience and skills of team members.

5.3 Ontologies of the Subject Area

In the actual Information Technologies (IT) field, it was of great importance to present knowledge of the subject area in a form suitable for interoperable use. Therefore, it was important that everyone using such knowledge equally and unambiguously understood its content. Such tools were provided by ontologies.

Ontology was a formal representation of knowledge on the basis of conceptualization, which described a set of objects and concepts and the relationships between them. An ontology formally consisted of terms organized into taxonomy or set of taxonomies, their definitions and attributes, associated axioms and inference rules. The set of assumptions constituting an ontology often took the form of a first-order logical theory, where dictionary terms were the names of unary and binary predicates, which were called concepts and relations, respectively. The simplest case was a taxonomy, which was defined as a special case of ontology, and described only a hierarchy of concepts connected by categorization relations. In more complex cases, appropriate axioms were added to reflect other relationships between concepts and limit their interpretation. Ontology was a knowledge base that described facts.

Many complex ontologies defined properties of domain concepts, and contained part of the knowledge of properties limiting the term meaning, which did not depend on another (changeable) part of the knowledge of this property. Such ontological knowledge was considered as a set of agreements about the subject area, and the other part of the knowledge of the property was a set of empirical and other laws of this domain. Therefore, the property ontology determined the agreement degree of term meaning between specialists in the subject area [23].

Connections between objects defined relations between concepts by property values. Unary connections were interpreted as properties of objects, and connections of arbitrary arity reflected various associations of objects. The modeling power inherent in binary relations made it possible to transfer any associative relations to properties, which made it possible to present PRs in the form of a network of related objects, which was known as a semantic network in the theory of knowledge representation. At the content level, the property ontology was a set of agreements (e.g., term definitions of the subject area, their interpretation, statements limiting the possible meaning of these terms, as well as interpretation of these statements), which were the result of an agreement between members of the community working in this property. An actual correspondence should be established between such properties of PRO as ontology, conceptualization, knowledge and validity, and the elements of this mathematical construction.

Property ontologies were used in IAO for:

- execution of complex information requests related to content processing at the semantic level (e.g., finding all projects carried out during a certain period of time by employees, who were in contact with employees of a certain division and had a certain level of authority, that is, had access to relevant corporate knowledge);

- interpretation of information on the experience and education of group members and other elements of their profiles;

- selection of tests and their adaptation to the specificity of the current task;

- integration of different terminology systems.

5.4 Competency Ontologies

The competency ontologies compared different types of information objects, such as vacancies, professions, training courses, etc., which were appropriate to take into account when assessing teams and their members. From the viewpoint of semantics, such information objects (IOs) were represented through sets of atomic competencies [24], and then the instances of different IO classes were compared by matching these values and evaluating their semantic proximity. All competencies were represented by sets of atomic competences (AC) with the following properties:

- $a \in C$, where, K is the set of IO class “Competency”, and $\mathrm{C}_{\text {attomic }}$ is the set of AC, i.e., $\mathrm{C}_{atom} \subset \mathrm{C}$;

- each competency was represented as an association of ACs:

$\forall c \in C, \exists a_i \in C_{\text {atomic }}, \mathrm{i}=1, \mathrm{n}, \mathrm{k}=\cup a$

- no AC was a subset of another AC: $\forall a, b \in C, a \subseteq b \Rightarrow b \notin$ Catomic

The set of AC properties caused only one set of atomic competencies for each competency $c \in C_{\text {there }}$. However, the problem of this approach lied in the need to use already built and formalized sets of competencies, which had a clear hierarchical order and were based on a generally accepted terminology system. Sources of such knowledge were various classifiers of qualifications and learning outcomes used in education and employment. Preference should be given to those, in which knowledge was presented based on an ontological approach, without additional operations for their formalization and semantic interpretation.

One of the well-known examples of competency ontology was an ontology of the multilingual ESCO [25] classifier, which defined and classified skills, competencies, qualifications and occupations. The main elements of ESCO were professions, skills and qualifications. ESCO described, defined and classified occupations, skills and qualifications relevant to the labor market and education and training in the United States.

ESCO was published by license of Linked Open Data, and used by developers in different formats: SKOS-RDF (with SKOS standing for Simple Knowledge Organization System and RDF standing for Resource Description Framework), and CSV (short name of comma-separated values) for applications that provided services, such as job search, career guidance and self-assessment. CSV was a simple text format designed for presenting tabular data.

The SKOS-RDF format was a W3C recommendation designed to represent knowledge models of PRs, such as thesauri, classification schemes, taxonomies, subject heading systems, or a structured controlled dictionary. SKOS was part of the Semantic Web family of standards built on top of RDF and RDFS, which aimed to enable the easy publication and use of such vocabularies as linked data. The RDF format, the resource description structure, was the language presenting ontologies.

Users integrated the ESCO classifier into their applications and services. In addition, ESCO provided a local application programming interface (API) and web service API so that applications and web services requested information from the classifier in real time. Competency ontologies were used for:

- formation of a group of executors or experts with sufficient knowledge and work experience to work in the area described in the form of a full-text document;

- determination of the formal specialty of the executor in a certain type of work;

- comparison of resumes and vacancies.

5.5 Organizational Ontologies

Researchers widely used three types of ontological models of organizations, which structured and organized information on their composition and properties:

- organizational ontology;

- People and Organizational (P&O) ontology of the organization's activities;

- ontology of user activities.

Organizational ontology reflected knowledge of the organizational and functional structure of a certain subject of economic activity, that is, its main components and the connections between them. This complex structure was often decomposed into separate hierarchical modules, and contained information on several aspects, such as the company's employees, the hierarchy of industrial relations between them, resources used in the production process in the enterprise, products (a consequence of the operation of the enterprise), structural units of the enterprise, and the connections between them [26]. Organizational ontology provided semantic information on the organization's structure.

The P&O ontology of the organization's activities was designed in order to organize and structure the functions and actions that took place in a certain P&O. Such a domain ontology of activities provided a hierarchical structure for classifying records documenting functions and activities for classifying and indexing goals.

The ontology of user activity was related to information seeking actions performed by end users. The task of information search was complex. From the perspective of knowledge-based analysis, it was necessary to determine which users needed to know what exactly about the IOs and actions used in the task of information retrieval, and how this knowledge should be organized. User actions were usually described through the actions they performed and the information objects associated with the information needs.

The development of the ontology of user activity began with the formation of a taxonomic classification of knowledge of the task and information objects. Task knowledge included a vocabulary to represent the process of performing actions, such as search, view, and save. All such three types of ontologies were used to somaticize IAO, but the most useful information was contained in organizational ontologies.

Such ontologies were used for:

- searching for a specialist responsible for a certain range of issues in the organization, regardless of what his position was called and how his job duties were formulated;

- identifying a group of employees of the organization exchanging data with each other for the implementation of a certain project, regardless of their belonging to different departments.

An important advantage of the proposed approach was that these ontologies were replaced without changes in their processing algorithms. For example, if it was necessary to evaluate a team from another educational institution or a team of a different type in general (e.g., participants of a research project from different institutions), it was enough to simply replace one organizational ontology with the other. In a similar way, it was possible to replace the ontologies of the PRs when transitioning to other types of team work and competency ontologies (e.g., when transitioning from a national structure of competencies to an international one). The use of ontologies and the knowledge gained from them were appropriate at all steps of IAO. As the first step of IAO, aggregation included composition of the team and providing hierarchical levels to team members:

- the organizational ontology allowed to determine the size of the team and the basic formal requirements for its members, thus comparing these requirements with the available candidates;

- competency ontologies were the basis for assessing the portfolio of team members, determining their real abilities, experience and skills, and analyzing the results of their formal and informal training;

- the ontology of PRs allowed resolving semantic ambiguities that may cause differences in terminology.

The second step of the IAO was providing an assessment task to be completed by team members:

- organizational ontologies determined which groups of tests should be selected for different team members according to their status in the team;

- the PRO allowed to choose the most informative and relevant types of testing in terms of both form and content;

- the competency ontology allowed to compare the test results with generally accepted classifiers of professions.

The third step of the IAO was related to the change of roles in the team:

- organizational ontologies made it possible to formalize the properties of roles in a team, thus determining their properties and the relationship between roles and their characteristics;

- competency ontologies allowed to formalize properties, define and take into account the semantic proximity between roles in the team (regardless of formal names, precisely by the set of necessary competencies and their level), and not only the hierarchical structure, which was the basis for more conditioned permutations;

- the PRO allowed interpretation of information on competencies and skills presented in the profiles of team members in various forms.

5.6 An Example of the HAA Semantics

The proposed approach to IAO semantics was considered using three ontologies:

- PRO “Armament”;

- organizational ontology “Military unit”;

- ontologies of ESCO competencies.

It should be noted that this was only a demonstration example, which only contained information from open sources, and aimed to show exactly how individual fragments of different ontologies were analyzed and used in all three phases of IAO. Real ontologies had a similar structure and the same expressiveness, but contained much more instances of different classes, for which all relevant values of data properties and object properties were defined. In this case, the task was to assess the readiness of the military unit to master a new type of weaponry. Such tasks were very relevant, and the existing competencies and experience of different teams needed to be evaluated and compared with the needs of new types of equipment. In contrast to traditional approaches, it was recommended to take into account not only the specialization of team members, their practical experience and ability to learn, but also more complex semantic similarities between already familiar weapons and new ones.

At the first step of IAO (aggregation), the team needed to be formed and the hierarchical levels of the team members should be determined. PRO determined the similarity between different types of weapons, and allowed to find semantically close instances (which parameters to take into account were explicitly determined at the same time) (Figure 3) and resolve semantic ambiguities that may cause differences in terminology.

Figure 3
Note: This figure was prepared by the authors.
Figure 4
Note: This figure was prepared by the authors.
Figure 5
Note: This figure was prepared by the authors.

At the same time, the semantic similarity between instances was evaluated by taking into account the length of the path between their classes in the class taxonomy and the value proximity of the selected data properties. Based on this, it was possible to determine which of the available teams should be evaluated further. The organizational ontology “Military unit” allowed to determine the composition and structure of such a team (Figure 4) and its subordination and components (Figure 5).

Figure 6. ESCO ontology structure
Note: This figure was prepared by the authors.

Competences of ESCO ontology (Figure 6) in this phase were a source of competencies, by which it was appropriate to evaluate the selected team, which were selected by similarity with existing ones and extended by their subclasses and atomic competencies). Ontology allowed to classify them by types (e.g., knowledge, skills, competencies), which were marked with yellow arcs by groups and means of presentation (natural languages) in Figure 6.

The second step of the IAO used ontologies to select assessment means of tasks and to analyze their results.

PRO allowed to find content-related tests and tasks, and align their terminologies with those used in the team. Therefore, the semantic relations of synonymy and “class-subclass” were used (marked with pink arcs in Figure 6) for the concepts of PRO. If this was not enough, it was recommended to determine the semantic proximity [27] between some concepts of PRO from this ontology in order to resolve the semantic ambiguity of both questions and answers.

Organizational ontology, which allowed to unambiguously determine the hierarchy of roles in the team, correctly defined the total assessment of the team according to the assessment of its members. It should be taken into account that different tasks for different team members may have different weights according to their roles.

ESCO's competency ontology provided a deeper interpretation of such results and supported a more complete use of information from group members' profiles, such as analysis of natural language portfolio elements for terms related to the concepts of PRs, obtaining more detailed information on the results of the education of team members from external knowledge bases, by deploying the acquired specialty into a hierarchically organized set of competencies.

- organizational ontologies defined exactly which groups of tests should be chosen for different team members according to their status in the team;

- the PRO allowed to choose the most informative and relevant types of testing in terms of both form and content;

- competency ontologies allowed to compare test results with generally accepted classifiers of professions.

At the third step of the IAO, the organizational ontology “Military unit” determined the admissibility of changing roles in the team: for military units with a clear and unambiguous hierarchy of team members, most of such changes were not acceptable and were performed only for team members with a status (rank) corresponding to a certain position.

The ESCO competency ontology helped determine which groups of competences led to a decrease in the overall result and which of the team members should improve their qualifications in these areas. These competencies were identified using information from the ontology of the “Armament” project.

It is important to note that it was enough to supplement the relevant ontologies or replace them with more relevant ones in the process of changing the assessment goals. It was only important that these ontologies were represented by OWL dialects, such as OWL Lite or OWL-DL, to ensure that the corresponding SPARQL queries were executed.

6. Discussion

The ontological approach to HAA semantics of teams, which were considered in the actual research work and this study, should become a component for integrated multi-criteria decision-making. First of all, the set of criteria itself was determined by the conditions of the task and was built on the basis of knowledge of PRO, but the relative importance of these criteria changed at different time points. In such an integrated approach, more complex collections of objects, which were connected by various semantic relations and belonged to different classes (e.g., groups of persons using different types of equipment and located in different places) needed to be evaluated, instead of homogeneous sets (e.g., teams consisting of IOs of the "Personnel" class).

The specificity of such approach integration was dynamism. Although the set of criteria for decision-making was generally determined by the user's task itself and defined levels of hierarchies, changes in the information environment affected the relative importance of these criteria. It was found that semantic similarity was an integral component of the analysis of complex IOs, and had three components: 1) attribute-based similarity, where the basis was information on the attributes of two concepts; 2) content-based similarity, which referred to the possibility of replacing two concepts without losing the meaning of their signification; and 3) similarity was based on distance. The smaller the semantic distance between two concepts, the closer their semantic similarity, and vice versa.

7. Conclusions and Further Research

In the perspective of further research, dynamic changes in the information space should be assessed using the assessment analysis of experts in the relevant sphere, and various points of view, criteria and domain rules were relevant for different levels of this task. Proposed models and methods aimed to evaluate teams with multi-level hierarchical structure using external ontological knowledge.

The knowledge necessary to solve the problem should be obtained from relevant various ontologies, which led to IAO expansion with means and methods of knowledge management and machine learning algorithms. Study of semantic similarity and proximity methods had limitations regarding classes and attributes in the ontology model, because it was unable to capture all the information on the concepts. Therefore, the considered concepts were also limited, because the development of a practical ontology model and inference system was a long and complex process, and the implementation in practice required repeated modification and assessment. As a result, further integrated research was needed to improve the ontological model and the lexical database, intelligent data analysis, machine learning and statistical data processing methods to provide even more accurate similarity assessments. Use of such metrics allowed to compare complex IO instances, taking into account the relative importance (hierarchy) of those criteria, which was significant from the viewpoint of the user's task at each current moment. It was assumed that the proposed approach was integrated with the semantic processing methods of big data metadata in order to search for information on the needs of complex IO users.

It is planned to work on adaptation of machine learning methods to acquire knowledge of level-specific rules and relations between levels in future. This knowledge can be acquired from experience of team assessments and real results of their work, retrieved from external ontologies, or be formalized from processing of expert opinions.

Author Contributions

Ontology theory, Atomic competencies analysis, Anatoly Gladun and Julia Rogushina; Hierarchical Aggregate Assessment theory, Martin Lesage; Text formatting in ATAIML template, Martin Lesage. All authors have read and agreed to the published version of the manuscript.” The relevant terms are explained at the CRediT taxonomy.

Data Availability

This scientific article was created within the framework of the state topic of fundamental research "Development of methods and means of integration of combinatorial optimization and semantic modeling for the intellectualization of information technologies", subject code VF 170.26 (2018-2023), and, partially, by the state topic of fundamental research “Methods and means of creating intelligent information and analytical systems using Semantic Web technologies, methods of machine learning and big data processing”.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

References
1.
F. L. Alemnge, L. C. Forje, N. A. Symphorien, B. Liudmyla, A. Avny, J. E. J.-E. Lane, A. Dan, C. Gomes, Z. Sohrabi, N. Zarghi, M. Lesage, G. Raîche, M. Riopel, F. Fortin, D. Sebkhi, and Y. Otis, New Horizons in Education and Social Studies. Book Publisher International, 2020. [Online]. Available: [Google Scholar] [Crossref]
2.
S. Ghosh, S. Thomke, and H. Pourkhalkhali, “The effects of hierarchy on learning and performance in business experimentation,” Acad. Manag. Proc., vol. 2020, no. 1, p. 20500, 2020. [Google Scholar] [Crossref]
3.
M. Lesage, “Hierarchical Aggregate Assessment (HAA): An assessment process of teams with several levels of hierarchy in education,” Creat. Educ., vol. 07, no. 14, pp. 1974–1994, 2016. [Google Scholar] [Crossref]
4.
G. Sammour, A. Al-Zoubi, A. Gladun, K. Khala, and J. Schreurs, “Semantic Web and ontologies for personalisation of learning in MOOCs,” in 2015 IEEE Seventh International Conference on Intelligent Computing and Information Systems (ICICIS) December 12-14, 2015, Cairo, Egypt, 2015, pp. 12–14. [Google Scholar] [Crossref]
5.
W. D. Nance, “Improving information systems students’ teamwork and project management capabilities: Experiences from an innovative classroom,” Inf. Technol. Manag., vol. 1, no. 4, pp. 293–306, 2000. [Google Scholar] [Crossref]
6.
H. Eysenck, Personality Theory and the Problem of Criminality. 1987. [Google Scholar]
7.
J. R. Graham, The MMPI: A Practical Guide. Oxford University Press, 1987. [Google Scholar]
8.
“Development of theoretical foundations for the modern computerized organization of the labor team,” 2021. https://www.sworld.com.ua/simpua17/sua17-1.pdf [Google Scholar]
9.
A. Y. Gladun and J. V. Rogushina, “Use of ontological analysis for evaluation of expert competences in the domain of national standards develoment,” Syst. Res. Inf. Technol., no. 3, pp. 19–32, 2016. [Google Scholar] [Crossref]
10.
O. V. Strokan, S. M. Pryima, J. V. Rogushina, A. Y. Gladun, D. V. Lubko, and A. A. Mozgovenko, “AdvisOnt: Semantization of Agricultural Advisory Services for Validation of Outcomes of Non-Formal and Informal Learning. I,” Control Syst. Comput., no. 1 (291), pp. 62–70, 2021. [Google Scholar] [Crossref]
11.
A. Hladun, Y. Rohushyna, and I. Subach, “Anontology modelling human resources management for innovational domains,” Collect. ”Inf. Technol. Secur.”, vol. 6, no. 1, pp. 15–25, 2018. [Google Scholar] [Crossref]
12.
A. Gladun, K. Khala, and R. Martinez-Bejar, “Development of object’s structured information field with specific properties for its semantic model building,” CEUR Workshop Proc. (ceur-ws.org), vol. 3241. pp. 102–111, 2022. [Google Scholar]
13.
M. Lesage, G. Raiche, M. Riopel, F. Fortin, and D. Sebkhi, The Internet Implementation of the Hierarchical Aggregate Assessment Process with the “Cluster” Wi-Fi E-Learning and EAssessment Application — A Particular Case of Teamwork Assessment. InTech, 2015. [Online]. Available: [Google Scholar] [Crossref]
14.
K. C. Laudon and J. P. Laudon, Management Information Systems: Organization and Technology. Prentice-Hall, Inc., 1995. [Google Scholar]
15.
K. Willey and M. Freeman, “Improving teamwork and engagement: The case for self and peer assessment,” Aust. J. Eng. Educ., vol. 12, no. 2, pp. 1–19, 2006. [Google Scholar]
16.
K. C Laudon, J. Laudon, and M. Brabston, Management Information Systems: Managing the Digital Frm. Pearson Canada, 2011. [Google Scholar]
17.
H. Van Zyl and L. Massyn, “Integrated assessment: A learning adventure and growth opportunity for adult learners,” Am. J. Bus. Educ., vol. 1, no. 2, pp. 95–104, 2008. [Google Scholar] [Crossref]
18.
R. W. Lingard, “Teaching and assessing teamwork skills in engineering and computer science,” J. Syst. Cybern. Inform., vol. 8, no. 1, p. 35, 2010. [Google Scholar]
19.
D. Gijbels, F. Dochy, P. Van den Bossche, and M. Segers, “Effects of problem-based learning: A meta-analysis from the angle of assessment,” Rev. Educ. Res., vol. 75, no. 1, pp. 27–61, 2005. [Google Scholar] [Crossref]
20.
A. Gladun and J. Rogushina, “Data mining – finding knowledge in data,” 2016. https://www.researchgate.net/publication/304025285_Data_Mining_Search_for_Knowledge_in_Data [Google Scholar]
21.
D. V. Lande, I. Y. Subach, and Y. E. Boyarynova, “Fundamentals of the theory and practice of intelligent data analysis in the field of cybersecurity,” 2018. https://ela.kpi.ua/bitstream/123456789/45721/1/NPOsnovy_teori_i_intelekt_analizu.pdf [Google Scholar]
22.
D. V. Lande and I. Y. Subach, “Visualization and analysis of network structures,” 2020. http://dwl.kiev.ua/art/vams/vams.pdf [Google Scholar]
23.
A. Gladun, J. Rogushyna, and M. Lesage, “A method for assessing the effectiveness of cybersecurity information and analytical support systems,” Coll. ”Inf. Technol. Secur., vol. 10, no. 2, pp. 1–13, 2022. [Google Scholar] [Crossref]
24.
J. Rogushina and S. Priyma, “Use of competence ontological model for matching of qualifications,” Chem. Bulg. J. Sci. Educ., vol. 26, no. 2, pp. 216–228, 2017. [Google Scholar]
25.
“The European multilingual classifier of skills, competences, qualifications and qccupations,” 2022. https://ec.europa.eu/esco/portal/home [Google Scholar]
26.
A. Gladun, K. Khala, and R. Martinez-Bejar, “Development of object’s structured information field with specific properties for its semantic model building,” CEUR Workshop Proceedings (ceur-ws.org), vol. 3241. pp. 102–111, 2021. [Google Scholar]
27.
A. Gladun and K. Khala, “Ontology-based semantic similarity to metadata analysis in the information security domain,” Probl. Program., no. 2, pp. 034–041, 2021. [Google Scholar] [Crossref]

Cite this:
APA Style
IEEE Style
BibTex Style
MLA Style
Chicago Style
Gladun, A., Rogushina, J., & Lesage, M. (2023). Hierarchical Aggregate Assessment of Multi-Level Teams Using Competency Ontologies. Acadlore Trans. Mach. Learn., 2(2), 55-65. https://doi.org/10.56578/ataiml020201
A. Gladun, J. Rogushina, and M. Lesage, "Hierarchical Aggregate Assessment of Multi-Level Teams Using Competency Ontologies," Acadlore Trans. Mach. Learn., vol. 2, no. 2, pp. 55-65, 2023. https://doi.org/10.56578/ataiml020201
@research-article{Gladun2023HierarchicalAA,
title={Hierarchical Aggregate Assessment of Multi-Level Teams Using Competency Ontologies},
author={Anatoly Gladun and Julia Rogushina and Martin Lesage},
journal={Acadlore Transactions on AI and Machine Learning},
year={2023},
page={55-65},
doi={https://doi.org/10.56578/ataiml020201}
}
Anatoly Gladun, et al. "Hierarchical Aggregate Assessment of Multi-Level Teams Using Competency Ontologies." Acadlore Transactions on AI and Machine Learning, v 2, pp 55-65. doi: https://doi.org/10.56578/ataiml020201
Anatoly Gladun, Julia Rogushina and Martin Lesage. "Hierarchical Aggregate Assessment of Multi-Level Teams Using Competency Ontologies." Acadlore Transactions on AI and Machine Learning, 2, (2023): 55-65. doi: https://doi.org/10.56578/ataiml020201
cc
©2023 by the author(s). Published by Acadlore Publishing Services Limited, Hong Kong. This article is available for free download and can be reused and cited, provided that the original published version is credited, under the CC BY 4.0 license.