Evaluating the Effectiveness of Commercially Developed Appraisal Instruments (CDAIs) Using Composite Indices to Assess, Compare, and Rank the Liveability, Quality of Living and Sustainability Performance of Cities and Communities

A large and diverse number of sustainability assessment tools, instruments, processes, and methodologies have been developed and are continuously introduced. The evolution of sustainability assessment is rooted in the necessity for properly embedding the principles of sustainability, capturing the notion of sustainable development, and addressing the stakeholders' needs, vision, and commitment to a better and sustainable future. Furthermore, the identification, selection, and design of indicators have become determinant processes in sustainability assessment; those processes aim to answer the questions of what to measure and how to measure progress in the implementation of sustainable development strategies, plans, programs, or policies.

The success of using the right set of indicators can be found in a group of commercially developed appraisal instruments (CDAls) that use composite indices to assess and certified the sustainability performance of a wide range of project types. Sustainability assessment rating systems successfully use indicators and composite indices in the construction built environment. With over 600 sustainability assessment rating systems developed worldwide [1] and billions of square foot construction and thousands of projects certified [2], the assessment methodology has gained recognition, popularity, acceptance, and credibility amongst the various groups of stakeholders [3], [4].

While every sustainability assessment rating system does not offer the same type of certifications schemes, some of the project types often achieving an above average ‘green’ or sustainability performance include new and exist- ing buildings, data centers, healthcare facilities, hospitality (e.g., hotel, motels, inns), retail, schools, warehouses and distribution centers, neighborhood development, and cities and communities. In the process of assessing the environmental and sustainability performance of the projects, LEED, BREEAM, Green Globes, CASBEE, Green Star and every other sustainability assessment rating systems identify, select, and design indicators which are then grouped in areas of performance, themes, or categories. These groups of indicators are meant to capture the notion of sustainable development and move the construction built environment toward a more sustainable future. Consequently, sustainability assessment rating systems focus on the identification, selection, and design of the set indicators that aims to balance the social, economic, and environmental performance of projects. While each sustainability assessment rating system has a distinctive scheme for each project type, some of the most common areas of performance, themes or categories in which the set of indicators are grouped include water efficiency, energy and atmosphere, transportation, materials and resources, indoor, environmental quality, waste, land use and ecology, pollution, and management. The assessment methodology also uses a weighting and aggregation system (W&AS) which is designed to indicate the relevance or importance of each indicator and area of performance, theme or category. Once the values are aggregated, the sustainability assessment rating sys- tem assigns a number of points to determine the level of performance achieved by the project after following a strict certification process.

While the previously described group of CDAls used rigorous processes to identify, selected, and design indicators, develop W&AS, and assure stakeholders of a specific 'green' or sustainability performance level through certification programs, there is a select group of CDAls using composite indices to take the process further to not only assess but also compare and rank sustainability performance of a specific type of projects. However, unlike the previous group, these CDAls do not offer certification programs. The second group of CDAls is widely known because of the regular publication of the score and rank achieved by participant cities and communities around the world. While the success of the first group of sustainability assessment rating systems has been proven, the use of the same methodology (i.e., composite indices) to compare and rank the sustainability, liveability, and quality of life of cities and communities around the world requires further examination. Three areas can be analyzed from the sustainability performance assessment standpoint: 1) the hierarchical level used in the assessment process, 2) the identification and selection of the elements (e.g., principle, sub-principles, criteria, indicators) included in each hierarchical level as a determinant factor in capturing the various facets of the sustainable development notion, and 3) the quantification methodology (i.e., W&AS) implemented by the developer or proponent of the assessment tool.

A wide range of commonly known CDAls uses a set of criteria and indicators to measure, benchmark, and rank the performance of cities and communities in areas aiming to capture various facets of the sustainable development notion. Liveability, quality of living, and sustainability are interrelated concepts; true sustainable cities and communities provide optimum liveable environments and satisfactory standards of quality of living for all residents. This research explored and drew conclusions on the effectiveness of incorporating the various facets of sustainable development into CDAls designed to assess, compare, and rank the quality of living, liveability, and/or sustainability of cities and communities around the world. The analysis is based on CDAls that are often published on a yearly basis (e.g., Global Liveability Ranking developed by the Economist Intelligence Unit [EIU]). CDAls are widely known, and their yearly publications reach an audience of millions; however, the development and use of composite indices raise a number of research inquiries in relation to how the sustainability assessment methodology is used by these group of CDAls:

(a) are composite indices an optimal assessment tool to measure progress toward the sustainability of cities?

(b) how effective are composite indices used by CDAls in capturing the notion of sustainable development?

(c) is it appropriate to compare the sustainability performance of cities using composite indices?

Diverse and robust scientific literature provides comprehensive and well-grounded criticism, reviews, examinations and analysis of CDAls. Previous efforts focus on four main areas: 1) individual analysis of the assessment approach used by the CDAI [5], 2) comparisons and analysis amongst CDAls [6], [7], [8], 3) broad issues concerning the effectiveness of CDAls in capturing 1, 2 more various facets of the sustainable development notion [9], [10], and 4) CDAls are part of comprehensive analysis aiming to understand and/or advance the concepts such as urban liveability, sustainability, and quality of living [11], [12]. Although CDAls have been studied from various perspectives, some characteristics remain to be investigated. Current literature does not take a closer look at nor provide the basis for a better understanding of the hierarchical structural organization (HSO) and the W&AS. More importantly, this study aims to close a knowledge gap by comparing three FDA against three commonly known CDAls to analyze the set of elements at each level of the HSO (sub-principles, criterion, indicators) and draw conclusions about the effectiveness of CDAls in capturing the various facets of the sustainable development notion. An analysis of CDAls using frameworks designed by consensus (FDC) as a basis of comparison is yet to be introduced. Similarly, there is a need for a closer critical examination of the HSO and W&AS applied by CDAls using composite indices to assess, compare and rank the quality of living, liveability, and sustainability of cities and communities.

The analysis methodology implemented to evaluate the effectiveness of CDAls using composite indices to capture the various facets of the sustainable development notion adopted a three-phase process:

First, the selection process to identify the three representatives, credible and reputable CDAls. Sustainability, quality of life, and liveability are three interrelated concepts that aim to capture the various facets of sustainable development in cities and communities. The identification and selection process of CDAls took into consideration a number of factors: a) although there is a wide range of CDAls, the three selected for the analysis were designed and introduced by different developers with excellent credibility and organizational reputation; b) each CDAI addresses one of the three topics under analysis (quality of living, liveability, sustainability); therefore, each CDAI represents those using in their name one of the three distinctive but correlated areas of performance contributing to the sustainable development of cities and communities; c) the assessment methodology and structure implemented by each CDAI allows the study of sustainability at various levels of the HSO typically implemented by assessment tools; d) the CDAls and their developers and proponents are widely known and add significant reputational value; e) based on the criteria and indicators included in each CDAI, the developers and proponents designed them adopting a holistic view of the issues at hand; and f) the three CDAls are published on a regular basis and reach millions of people around the world. Using the six factors previously listed, the three CDAls using composite indices selected for the analysis were the EIU's Global Liveability Index and Ranking, Arcadis' Sustainable Cities Index, and Mercer's Quality of Living Survey and Ranking.

Second, the selection process to identify FDC by groups of multi- and inter-disciplinary stakeholders. The credibility of the developing organization, transparency and wellstructured selection process of elements (e.g., criteria, indicators), and an effective engagement and participation of various and large stakeholder groups were the three main factors in the identification and selection process of FDC. The three frameworks selected for the analysis were ISO 37130:2018 Sustainable cities and communities Indicators for city services and quality of living, United Nations Sustainable Development Goals (UN SDGs)-Goal 11: Sustainable Cities and Communities, and customized frameworks for sustainable cities (CFSS) developed and implemented by the cities of Vancouver and Montreal.

Third, the evaluation of CDAls effectiveness process investigated the (1) HSO, (2) set of sub-principles, criteria, and indicators, and (3) quantification methodology (i.e., W&AS) implemented by CDAls. The evaluation of CDAls was performed on four of the five hierarchical levels: principle (i.e., composite index), sub-principle (i.e., composite sub-index), criterion (i.e., area of performance, theme, or category), and indicator. The set of sub-principles, criteria, and indicators implemented by CDAls were compared (i.e., benchmarked) using the three FDC. Similarly, the W&AS adopted by CDAls to measure the relevance of sub-principles, criteria, and indicators was included in the analysis of the effectiveness of these tools in incorporating the different facets of the sustainable development notion.

The terms ranking, scores, index, surveys, lists, and other methodologies using composite indices developed with the aim of comparing and ranking the performance of cities and communities are often used interchangeably. Although it is a diverse and crowded field, common themes can be identified amongst the existing CDAls. Considering factors such as assessment methodology, areas of performance (i.e., criterion), and indicators used, CDAls using composite indicators can be grouped into sector or topic-based, governmental law or policy decision making-based, and organization or cost of living-based. The sector or topicbased category includes CDAls designed to focus on or target a specific group. Amongst many others, some topicbased CDAls include Monocle's Most Liveable Cities Index, Forbes' Best Places to Retire, QS' Best Student Cities, Forbes' Best Places for Business and Careers, Money's Best Places to Live, Forbes' Best and Worst Places for Job Growth, MoneySense's Canada's Best Places to Live, and MoneySense's Best Places to Raise Kids. The governmental law or policy decision-making-based category aims to assist the decision-making and/or policy-making processes by including indicators reflecting the concept of quality of living. Most of the CDAls in this category are supported by or developed in collaboration with academic institutions or non-profit organizations. Some international raking in this category includes the GreenScore City Index developed by GreenScore Canada, the Martin Prosperity Institute's Most Livable Canadian Cities or Livability Top 100 Places to Live, the Lee Kuan Yew School of Public Policy's Global Liveable Cities Index, and the Ease of Living Index (EoLI) published by the Ministry Housing and Urban Affairs (MoHUA), Government of India. The organization of cost of living-based rankings is mostly designed to support organizational or business decision-making processes. The CDAls provide support to organizations, business leaders, and human resources professionals in the tasks of opening or relocating businesses, and paying compensation to expatriates or business travellers. Some of the CDAls included in this category are Mercer's Quality of Living Index, EIU's Green City Index commissioned by Siemens, Employment Conditions Abroad (ECA) International's Location Rating for Expatriate Living Conditions and EIU's Global Liveability Ranking.

The EIU developed and publishes the Global Liveability Index and Ranking annually. Using 30 qualitative and quantitative factors (i.e., indicators) grouped into 5 categories (i.e., criteria), “The Economist Intelligence Unit’s liveability rating quantifies the challenges that might be presented to an individual’s lifestyle in 140 cities worldwide” [13]. The assessment methodology evaluates each city in the categories of stability (5 indicators), healthcare (6 indicators), culture and environment (9 indicators), education (3 indica- tors), and infrastructure (7 indicators). The rating assigned to qualitative indicators is based on the judgment of in- house expert country analysts and a field correspondent based in each city (i.e., in-city contributor), whereas quantitative indicators receive their rating based on comparison performance of the location using external data points [14]. Qualitative and quantitative indicators in each category are scored as acceptable, tolerable, uncomfortable, undesirable, or intolerable. Each weighted category is “equally divided into relevant subcategories to ensure that the score covers as many indicators as possible” [13], [14]. Moreover, indicators within each category are equally weighted, which is interpreted as equal relevance for each indicator within a category. The average of the indicators within a category is weighted, and the sum of all categories is the city’s overall rating. The total weight of all five categories adds up to 100%: stability, 25%; healthcare, 20%; culture & environment, 25%; education, 10%; infrastructure, 20%. Once the categories are weighted, the final rating indicates the liveability of a city. A rating of 100 means the liveability in the city is ideal, whereas the liveability of the city turns more intolerable as the rating score approaches 1 [13], [14].

The Arcadis' Sustainability Cities Index was based on research produced by the Center for Economics and Business Research Ltd (Cebr). The design and consultancy firm for natural and built assets, Arcadis, wanted to explore how 100 of the world's leading cities are doing across three areas of performance: people, planet, and profit. These represent the social, environmental, and economic pillars of sustainability. The index was developed with a holistic perspective in which people, planet, and profit are amalgamated to provide a better understanding of each location (i.e., city) and its position on the sustainability scale. Various areas of assessment (i.e., criteria) were included in each of the three sub-indices. Those criteria included in the people (i.e., social pillar) sub-index are meant to capture the present levels of 'quality of living' in the city while those in the planet (i.e., environment pillar) sub-index intent to represent the 'green attributes' of the city and criteria in the profit (i.e., economic pillar) sub-index aim to reflect the city's 'economic health.' Each sub-index is decomposed into a number of criteria, and in some instances, the criterion is the result of integrating two or more indicators. To capture the "quality of living" in a city, the Arcadis' Sustainable City Index uses 17 indicators grouped in 13 criteria, environmental performance is evaluated using 16 indicators in 11 different criteria, and the city's "economic health" is represented by seven criteria encapsulating 15 indicators [15]. Most criteria are composites, meaning these are the average of their component indicators. Criteria within each sub-index have a specific weight, and all criteria within a sub-index add to $100 \%$. Once each sub-index is known, the city's overall score is equal to the average of the three sub-indices.

In 2019, Mercer, an American global consulting leader in talent, health, retirement, and investments, released its 21 st Quality of Living Survey and Ranking. Due to the effects of the COVID-19 pandemic, Mercer did not release its annual survey or ranking in 2020 . The Quality of Living Ranking is developed using Mercer's Worldwide Quality of Living Survey. Although the latest report ranks 231 cities worldwide, Mercer's Quality of Living Ranking provides hardship premium recommendations for over 450 cities [16]. These recommendations aim to support organizations on two specific fronts: fair compensation of employees when organizations are relocating them on international assignments and knowledge-based decisions for multinational organizations opening open offices or plants [16]. The city's quality of living is not only an important variable for employers and expatriates but also for leaders and municipalities seeking to set up new businesses. Therefore, it is critical to identify, understand, and improve the performance of those specific factors affecting the residents' quality of living. Mercer clusters 39 factors (i.e., indicators) in 10 categories (i.e., criteria). Cities are compared to a base city (New York City) which is assigned a base score of 100 . In addition to the overall Quality of Living Ranking, Mercer also allows comparing cities across areas of performance. Attributing a score to each factor allows an objective city-to-city comparison; factors are weighted to reflect their relative importance to expatriates. The quality of living index resulting from the comparison of relative differences between two locations allows linking such an index to a quality of living allowance amount. Furthermore, "for the indices to be used effectively, Mercer has created a grid that enables users to link the resulting index to a quality of living allowance amount by recommending a percentage value in relation to the index" [16].

Several other CDAIs, including rankings, scores, indices, lists, and surveys, can be found around the world. One or more facets of sustainable development have an impact on the assessment methodology implemented by the 50 CDAls included in Table 1. Each city's performance is assessed and compared amongst the participant cities, and the results are typically available on a yearly basis; the list in Table 1 is not meant to be comprehensive but rather provide a sample of the diverse CDAls available to better understand the quality of living, liveability, and sustainability performance of cities and communities around the world. Some CDAls developers and proponents establish their own set of indicators and criteria, whereas others use and combine the outcomes of reputable surveys. For instance, the World's Best Cities to Live Index published by Global Finance Magazine is developed using three of the best cities lists: EIU's Global Liveability Index, Mercer's Quality of Living Ranking, and Monocle's Most Liveable Cities Index [17]. The reputation of several CDAls is based on (1) credible assessment methodology, (2) continuous, periodic, and reliable publication, and (3) openness to improvement and adaptation. The use of CDAls to evaluate, compare and rank the quality of living, liveability, and sustainability performance of cities and communities has transcendent to other scenarios. With a focus on sustainability, CDAls are often used at the organizational and country levels to assess the performance of one or various facets of the sustainable development notion. As a manner of example, Corporate Knights publishes the World's 100 Most Sustainable Corporations and America's Most Responsible Companies released by Newsweek Magazine aim to rank companies based on their sustainability performance. Similarly, the UN SDGs Index and Dashboards to track the progress of each country towards the adoption of the 17 SDGs, the Human Development Index (HDI) developed by Mahbub ul Haq and adopted by the United Nations Development Programme were replaced in 2010 by the Inequality-adjusted Human Development Index (IHDI), and the Better Life Index (BLI) created by the OECD intents to compare countries' performance based on preferences of what makes for a better life.

The three CDAls previously discussed in detail were selected to analyze the effectiveness of using composite indices to capture the various facets of the sustainable development notion. The EIU's Global Liveability Ranking, Arcadis' Sustainable Cities Index, and Mercer's Quality of Living Survey and Ranking use indicators to assess, compare, and rank the sustainability, liveability, and quality of living performance of the participant cities and communities around the world. On the other hand, the three FDC selected to analyze the appropriateness of the set of subprinciples, criteria and indicators used by CDIAs were ISO 37120:2014, UN SDGs - SDG 11, and CFSS developed and implemented by the cities of Vancouver and Montreal. The analysis of CDAls also included two other areas of performance. Observations were formulated and articulated in relation to 1) the HSO used to set the logical organization of the set of elements used in the assessment (i.e., sub-principles, criteria and indicators) and 2) the W\&AS implemented to calculate the stand-alone index.

CDAls and FDC have embedded an HSO. While CDAls organize the various sub-principles, criteria, and indicators to facilitate the assessment and management (e.g., understanding and communication of outcomes), proposing an HSO is not the main goal of FDC. FDC do not weigh or aggregate the values of elements to calculate a stand-alone number (i.e., composite index). Nevertheless, FDC use the characteristic of the HSO of organizing a set of elements (e.g., indicators) by themes, categories or areas of assessment which are the common names given to the above level in the hierarchy but formally known in sustainability assessment as criteria. Table 2 includes the different hierarchical levels commonly found in CDAls and FDC. Arcadi's Sustainable Cities Index is the only CDIAs using the hierarchical level 'sub-principle' to aggregate the criteria in 3 different pillars: people (social pillar), planet (environment pillar), and profit (economic pillar). The hierarchical level 'criterion' is included in all 3 CDAls, and 2 of the FDC are included in the analysis. The structure followed by the UN in the design of its SDGs does not group the indicators included in each SDG under a specific criterion, neither the hierarchical level 'criterion' was identified in the two plans developed by the City of Montréal.

Another contentious observation besides the HSO is the terminology found in CDAls and FDC. The area of sustainability assessment uses specific notations and definitions for each element of the HSO. However, CDAls and FDC use terms interchangeably and, in some instances, wrongly. Table 2 lists the different terms found in CDAIs and FDC included in the analysis. Criteria are called categories, areas of assessment, factors, themes, goals, or sectors, whereas terms to denominate an indicator include factor, target, and sub-category. The use of misleading terminology extends to the overall name of the CDAls. CDAls use composite indices to assess, compare and rank the performance of cities and communities. Therefore, the CDAls do not present an index because the performance score of each assessed city or community does not contribute to an overall number. Once the performance of each city or community is assessed, the results are compared and ranked from best to worst. As a result, ranking instead of score, index, list, or survey is the technically correct term to name any CDAls using composite indices to assess, compare and rank performance.

CDAls and FDC also contribute to solidifying the connection among the quality of living, liveability, and sustainability. While the EIU uses the simplistic view of liveability to "assess which locations around the world provide the best or the worst living conditions" [13], the EIU's Global Liveability Ranking is described as a CDAI that ranks cities and communities on their urban quality of living. 9 out of the 30 qualitative and quantitative factors specifically measure the 'quality of' something. Arcadis aims to assess, compare and rank sustainability performance; however, the concept of quality of living is not only behind the rationale of several criteria and indicators but also has a central role in the design of the 'People sub-index' which "measures social sustainability-quality of living in the present and prospects for improvement for future generations" [15]. Similarly, ISO suggests a clear connection between sustainability and quality of living. ISO 37120:2018 standard focused on "city services and quality of life as a contribution to the sustainability of the city" [18]. On the other hand, the UN SDGs-Goal 11 and CFSS (e.g., Sustainable Montréal 20162020 Plan, Greenest City Action Plan [GCAP], Climate Plan 2020-2030) use sustainable development strategies to provide a better quality of living and healthy and acceptable liveable environments in cities, communities and human settlements [19], [20], [21], [22].

Because the principle and sub-principles guide the primary framework for managing the quality of living, liveability, and sustainability of cities and communities, the hierarchical levels of 'principle' and 'sub-principle' provide justification for the use of criteria and have a critical role in the assessment and management of performance. To go from a large number of potential indicators to a handful of sub-principles in an organized and understandable manner, intermediate levels of the $\mathrm{HSO}$ aggregate qualitative and quantitative information and data. The hierarchical level 'sub-principle' is often used by CDAls and FDC to group elements in the hierarchical level below (i.e., criterion), as noted in Table 2. Although several dimensions of sustainability have been proposed [23], [24], [25], [26], [27], the triple bottom line remains the most widely known and accepted sustainability framework [28], [29]. These three distinctive but interconnected and interdependent domains, which are often referred to as pillars or dimensions, intend to capture and link the various facets of sustainability (i.e., environmental protection [planet], economic viability [profit], social equity [people]). The use of the hierarchical level 'sub-principle' in sustainability assessment is not arbitrary; it allows the easy identification and classification of criteria and indicators and facilitates the implementation of the assessment, weighting and aggregation processes.

The criterion is the third hierarchical level. The analysis concluded that there is not framework outlining the identification and selection criteria to calculate a composite index to assess the quality of living, liveability, or sustainability of cities and communities; therefore, the number of criteria and the areas of performance the criteria aim to assess are defined by the CDAI's developer or proponent. Nevertheless, the use of the hierarchical level 'criterion' is more prevalent than the hierarchical level 'sub-principle' in CDAls and FDC. All three CDAls and two of the four FDC group indicators using welldefined criteria; however, the type (i.e., areas of assessment) and the number of criteria included in CDAIs and FDC differ significantly. The EIU's Global Liveability Ranking, Arcadis' Sustainable City Index, and Mercer's Quality of Living Ranking include the hierarchical level 'criterion' using 5 categories, 31 areas of assessment, and ten categories, respectively. In contrast, the hierarchical level 'criterion' is completely absent or its identification encounters challenges in some of the FDC used in the analysis. UN SDGs-Goal 11 does not use the hierarchical level 'criterion', whereas the Climate Plan 2020-2030 designed by the City of Montréal includes 46 actions grouped in five sectors, but progress is measured using a mere handful of indicators. The Climate Plan 2020-2030 implemented by the City of Montréal was preceded by Montréal's First Strategic Plan for Sustainable Development 2005-2009, Montréal's Corporate Sustainable Development Plan 2010-2015, and Sustainable Montréal 2016-2020. The Climate Plan 20202030 includes the following five sectors: 1) mobilization of the Montréal community; 2) mobility, urban planning and urban development; 3) buildings; 4) exemplarity of the city; and 5) governance [30]. To monitor progress, the City of Montréal publishes an annual report on the Climate Plan based on eight indicators grouped into the reduction of GHG emissions (4 indicators) and resilience/adaptation (4 indicators).

The two FDC that include an easily identifiable hierarchical level 'criterion' are ISO 37120:2018 (19 themes) and the GCAP designed by the City of Vancouver (10 goals). The GCAP is the latest available plan related to Vancouver's urban sustainability. The GCAP includes 10 goals with their respective indicators and targets plus an additional goal related to greening the city's operations [21]. The 10 goals included in the GCAP are: 1) climate and renewables; 2) green buildings; 3) green transportation; 4) zero waste; 5) access to nature; 6) clean water; 7) local food; 8) clean air; 9) green economy; and 10) lighter footprint. Furthermore, the GCAP is supported by other plans and strategies including but not limited to Climate Emergency Action Plan, Zaro Emissions Buildings Plan, Zero Waste 2040, Climate Change Adaptation Strategy, and Renewable City Strategy [20].

The diversity of approaches in the use of the hierarchical level 'criterion' resulted in limited similarities between CDAls and FDC. Two types of criteria were identified to facilitate the analysis and easy understanding of the similarities between CDAls and FDC at the criterion level of the HSO. Criteria categories I and II are described as:

- Criteria category I: Criteria with the same name included in both the CDAls and FDC

- Criteria category II: Criteria which names in CDAls and FDC are not the same but resemble some similarities in the area of performance to be assessed.

Table 3, Table 4 and Table 5 summarize the comparison between CDAls and the four FDC included in the analysis. A comparison between the criteria proposed by ISO 37120:2018 and all three CDAls provides the following observations on criteria category I: 1) the ElU's Global Liveability Ranking includes two, health and education; 2) Arcadis' Sustainable Cities Index includes three, education, health and energy; and 3) Mercer's Quality of Living Ranking includes two, safety and recreation. Regarding criteria category II, the similarities are more prominent. The number of common criteria category II between ISO 37120:2018 and each CDAI is seven with Mercer's Quality of Living Ranking, ten with Arcadis' Sustainable Cities Index, and one with the EIU's Global Liveability Ranking. Similar analysis between CDAls and the GCAP designed by the City of Vancouver indicated that (1) there are no criteria category I or II similar between the EIU's Global Liveability Ranking and the GCAP, (2) there are no criteria category I similarities between the GCAP and Arcadis' Sustainable City Index or Mercer's Quality of Living Ranking, (3) eight criteria category II are included in Arcadis' Sustainable Cities Index and the GCAP, and (4) three criteria category II are included in Mercer's Quality of Living Ranking and the GCAP.

Indicators infer attributes of the quality of living, liveability, or sustainability of cities and communities. CDAls and FDC identify and select a set of indicators that are capable of transferring valuable qualitative and quantitate information. Moreover, the information captured by each indicator aims to convey a single meaningful message regarding the different facets of sustainable development. To facilitate the analysis of the hierarchical level known as 'indicator', comparisons between CDAls and FDC have been summarized in Table 3, Table 4, and Table 5. Indicators were classified in indicators category I and category II to facilitate the analysis and easy understanding of the similarities between CDAls and FDC at the indicator level of the HSO. Indicators category I and II are described as:

- Indicators in category I: The same or analogous indicators included in both the CDAIs and FDC

- Indicators category II: Indicators in the CDAls that aim to capture some facets of the area of performance addressed by the indicator(s) included in the FDC.

The large number of indicators proposed by ISO $37120: 2018$ created more opportunities to find common indicators type I and II with each CDAI included in the analysis. Considering that ISO $37120: 2018$ proposes 128 indicators ( 45 core, 59 supporting, and 24 profile), only one indicator type I (0.78%) proposed by ISO 37120:2018 was found in the EIU's Global Liveability Ranking, which is part of category 5 , infrastructure. Similarly, 18 indicators type II (14.4%) used by the ElU's Global Liveability Ranking capture some of the areas of performance included in ISO $37120: 2018$. While there is an improvement in indicators type I, fewer indicators type II were common between the other 2 CDAls and ISO 37120:2018. Arcadis' Sustainable City Index includes 13 indicators type I (10.16%) whereas Mercer's Quality of Living Ranking uses seven indicators type I (5.5%). 11 (8.6%) and 6 (4.7%) indicators type II were found in Arcadis' Sustainable City Index and Mercer's Quality of Living respectively.

The UN [31] recognizes "the cross-cutting nature of urban issues, which have an impact on a number of other Sustainable Development Goals, including SDGs 1, 6, 7, 8, 9, 12, 15, and 17 , among others." Therefore, two other categories of indicators were used in the analysis. Following the same reasoning of indicators category I and II, indicators category I-a and category II-b are indicators included in UN SDGs other than Goal 11. Considering the 15 indicators listed under UN SDG-Goal 11, only one indicator type I was found in one of the three CDAls. Arcadis' Sustainable City Index and UN SDG-Goal 11 have in common the indicator 'mean level of pollutants (particular matter). The inclusion of indicators category II is also limited. UN SDGs-Goal 11 has in common three indicators category II with the EIU's Global Liveability Ranking, whereas 5 and 6 indicators type II were found in Arcadis' Sustainable City Index and Mercer's Quality of Living, respectively. Further analysis of other UN SDGs found a number of indicators, category I-a and category II-b, aligned with indicators used by CDAls. The most common indicators between the FDC (i.e., UN SDGs) and the CDAI were category II-b which refers to an indicator(s) in the CDAI capturing some facets of the area of performance assessed by the FDC. Indicators used by the EIU's Global Liveability Ranking were found in UN SDGs 3, 4, 5, 6, 7, 9, 11, 13, 16, and 17. Similarly, Arcadis' Sustainable City Index includes indicators found in UN SDGs 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 16, and 17 , whereas UN SDGs 3 , 4, 5, 6, 7, 8, 10, 11, 16, and 17 includes indicators used by Mercer's Quality of Living Ranking.

The analysis also identified 21 indicators implemented by the City of Vancouver while the City of Montréal designed various targets and actions but only tracks performance using eight indicators. The EIU's Global Liveability Ranking and the City of Vancouver and the City of Montréal have in common a limited number of indicators type II. There is an improvement in the number and type of indicators used by Arcadis' Sustainable City Index in relation to those implemented by the City of Vancouver and the City of Montréal. Nevertheless, most of the indicators are included under the planet pillar, and only one indicator category II was found in common between the Arcadis' Sustainable City Index and the City of Vancouver. The communalities of indicators type I and II between Mercer's Quality of Living Ranking and both cities are also limited. Three indicators type I, and four indicators type II were found in Mercer's Quality of Living Ranking and the City of Vancouver, whereas one indicator type I and five indicators type II were common between Mercer's Quality of Living Ranking and the City of Montréal.

The W&AS assigns relevance (i.e., weight) to each element in reference to other(s) in each level of HSO and consolidates performance results in order to calculate a composite index. Because of the lack of a standardized W&AS framework, CDAls use a variety of weighting and aggregation approaches. Figure 1 illustrates the three most common W&AS approaches implemented by CDAls using composite indices to evaluate the quality of living, liveability, and sustainability of cities and communities. Elements within each intermediate hierarchical level can be weighted or averaged (i.e., equally weighted); equal weight indicates that elements are considered by stakeholders as equally relevant or have equal performance impact (e.g., social, economic, or environmental). Commonly, sub-principles (i.e., pillars or dimensions) are averaged, criteria are weighted, and indicators are either averaged or weighted based on the approach adopted by the developer or proponent of the CDAls. Furthermore, CDAls often develop and rely on internal methodologies for not only the identification and selection of sub-principles, criteria, and indicators but also the design of W&AS to be used in the assessment, comparison, and ranking of performance. Independently of the W&AS, the summation of the weights of each element within a certain level of the HSO must equal 100% independently of the approach implemented by CDAls developers.

The most common approach is a combination of qualitative and quantitative indicators to measure progress on the implementation and performance of sustainable development strategies designed to improve the quality of living, liveability, and sustainability of cities and communities. Indicators can be further decomposed into sub-indicators which add another level to the HSO. The source of data for each indicator can be among other approaches an in-house 'rating' (e.g., 'EIU rating'), external data (e.g., The World Bank, UNESCO, World Resources Institute) and other CDAls (e.g., composite indices [e.g., Arcadis' Sustainable City Index uses Siemens Green City Index to assess its indicator 'Green space as % of city area']). Arcadis' Sustainable City Index, the EIU's Global Liveability Ranking, and several other CDAls use a simple average for the hierarchical level 'indicator' but assign a different weight to each criterion unless otherwise specified (e.g., The affordability criteria included in Arcadis' Sustainable City Index is weighted 70:30). The bottom illustration in Figure 1 describes the mixed approach of the W&AS used by Arcadis' Sustainable City Index and the EIU's Global Liveability Ranking. Other CDAls use the top-down approach to distribute equal weight to elements within the hierarchical level 'sub-principle' and 'criterion', but the number of elements within the hierarchical level 'indicator' has a crucial role in determining the weight assigned to each element. Instead of allowing the number of indicators to determine the weights, the developer or proponent of the CDAls may choose to assign specific weight to each indicator and then roll up the values to calculate the composite index; however, this bottom-up approach often requires the use scientific-based methodologies to support the decision-making process of weights allocation.

CDAls use a bottom-up (i.e., roll-up values) approach to calculate the final composite index. Scores at the hierarchical level 'indicator' are converted into common units. CDAls often use percentages, in-house ratings, or external standard performance as a parameter of comparison or reference (e.g., New York City is often used as a baseline score or control city to compare the performance of other cities or assign performance points based on a predetermined rating). After averaging or weighting the performance at the hierarchical level 'indicators', the values of criteria are also averaged or weighted to then aggregate typically three scores at the hierarchical level 'sub-principle' (i.e., dimensions or pillars) in order to calculate the final composite index. The general consensus amongst CDAls using composite indices is averaging the elements at the hierarchical level 'sub-principle' as an indication of the equal relevance (i.e., weight) between the sets (i.e., dimensions or pillars) included in the sustainable development notion.

The analysis conducted identified a diverse number of performance factors that determine the effectiveness of CDAls using composite indices as a procedural methodology to assess, compare, and rank the quality of living, liveability, and sustainability of cities and communities. The effectiveness of CDAls is often evaluated using a unidimensional approach focusing on the areas of performance (i.e., indicators) that properly capture the various facets of the principles (e.g., sustainability, liveability, quality of living). A more holistic approach identified a wide range of issues and potential areas of improvement. Although the research primarily focused on $3 \mathrm{CDAls}$, some of the others included in Table 1 were studied as part of the process of understanding the application of composite indices to assess, compare and rank the performance of cities. The issues identified in the analysis provide opportunities for improvement and future research to overcome the current management and assessment challenges to capture the various facet of the sustainable development notion encountered in the approaches used by CDAls including but not limited to the identification and selection of elements in each level of the HSO and the W\&AS methodologies.

The following issues, which also provide opportunities for improvement and future research, were identified in the analysis of the design, development and implementation processes (e.g., identification and selection of elements [e.g., criteria, indicators], data collection) of CDAls along with the HSO, set of sub-principles, criteria and indicators, and assessment methodology (i.e., W\&AS) implemented by proponents and developers:

a) Engagement and participation of stakeholders: Stakeholders and decision-makers engagement and participation facilitate the acceptance of outcomes and validate the assessment methodology by bringing accountability and credibility to the process. Meeting the needs and satisfyingto a reasonable extent-the expectations of stakeholders is a prerequisite to effectively implementing sustainable development strategies and subsequently accomplishing sustainability [28]. Because their input reflects their needs and expectations, the most relevant group of stakeholders in the assessment of the quality of living, liveability, and sustainability of cities and communities are their inhabitants. While FDC engage a wide range of interdisciplinary stakeholders and decision-makers, CDAls are designed by a limited group of in-house experts and consultants.

b) Identification and selection of elements in each level of the HSO: A lack of a standardized framework for the identification and selection of sub-principles (i.e., dimensions or pillars), criteria and indicators give flexibility to developers and proponents of CDAls. How many and which ones should be part of the assessment process are still two of the main areas of debate in sustainability assessment studies. Finding an agreement on dimensions, criteria and indicators is linked to defining sustainability which is one of the main obstacles to overcome because of the continuous evolution of the term. Nevertheless, although there is not a standardized methodology for the identification and selection of sub-principles, criteria, and indicators, credible and well-developed FDC can support the design process of CDAls.

c) Weighting sub-principles, criteria, and indicators: CDAls using composite indicators assign weights to subprinciples, criteria and indicators. The weighting process also lacks a standardized framework. Developers or proponents of CDAls choose between assigning the same or different weights to each element within a hierarchical level. While assigning equal weight to sub-principles (i.e., dimensions or pillars) and indicators and different weights to criteria is a common practice, it does not necessarily reflect the latest advances in sustainability as a concept and assessment methodologies. Moreover, developers and proponents of CDAls using composite indicators do not often elaborate on the methodologies used or the reasoning behind assigning equal or different weights to elements.

d) Disguising actual performance using quantity or weight of elements as determining factor: The composite index assessment methodology can be designed with the intent of hiding or embracing elements with low or high performance. Based on the number or weight of the elements, the city or community can achieve high scores. CDAls developers or proponents may choose to include a certain number of elements within a hierarchical level or assign specific weight to elements in a manner that poor areas of performance are hidden or embraced. For instance, a city or community with a low or high-performance score in a specific area may end up at the top of the ranking because the influence of that element is either dissuaded or embraced.

e) Quantity and type (i.e., areas of assessment) differ amongst CDAls-ls it uniqueness or manipulation? The numbers of elements (i.e., sub-principles, criteria, indicators) differ amongst CDAls using composite indices to assess performance. Regarding the quality of living, liveability, and sustainability concepts, the discrepancy in quantity and type of elements within each hierarchical level confirms (1) the lack of a rigorous theoretical framework, (2) the continuous evolution of the concepts, (3) a widely agreed definition of the concepts have not emerged, and (4) "people define sustainability in the ways that suit their particular applications, oftentimes with no explicit evidence and recognition of the exact meaning being implied" [32].

f) Selecting Meaningless Indicators: Effectively capturing the different facets of the sustainable devilment concepts depends on selecting meaningful criteria and indicators. The inclusion of an indicator must consider the ability of the city to have control over the area of performance, the impact of that area of performance on the inhabitants, and the contribution of the area of performance to achieve a better quality of living, liveability and sustainability. In addition to those failing those three considerations, some indicators used in the 3 CDAls included in the analysis are not part of or proposed by any of the 3 FDC. The 3 CDAls included in the analysis include indicators such as the number of 'things to do' on TripAdvisor, climate, alcoholic beverages, and humidity/temperature rating, among several others.

g) Assigning equal weight to elements is misleading, and the easy way out: CDAls commonly assign equal weight to elements within the same hierarchical level (i.e., sub-principles, criteria, indicators). Although some CDIAs argue that elements within a hierarchical level are not equally weighted (i.e., indicators) because the elements within the hierarchical levels are weighted (e.g., criteria), assigning equal weights to elements under a criterion or sub-principles does not reflect stakeholders' needs and expectations, actual [social, economic, environmental] impact of the elements, or relative relevance of an element in reference to others. An equal weight of elements wrongly indicates that each one of them has the same degree of contribution to advance the quality of living, liveability, and sustainability.

h) CDAIs are using outdated elements to assess performance: Although rankings and other instruments designed to assess the performance of cities continue to emerge, well-known CDAls using composite indices have been in the market for a long time. Not only the concepts of quality of living, liveability, and sustainability are dynamic and continuously evolving, but also FDC have been developed in the last few years. ISO $37120: 2018$ was first published in 2014, UN SDGs were set up in 2015 , and sustainability city plans are designed and published periodically. Making CDAls living or evergreen assessment instruments would allow the consideration and inclusion of the latest scientific developments and FDC proposals.

i) Assessing versus comparing and ranking performance of cities and communities: CDAls use composite indices to assess, compare and rank performance. While the technique itself faces challenges to accurately capture the various facets of the principle and assess performance, comparing and ranking the quality of living, liveability, and sustainability performance of cities and communities around the world comes with an additional obstacle. The need and expectations of stakeholders vary around the world; therefore, the meaning of quality of living, liveability, and sustainability has a geographical component. Cities and communities assessed, compared, and ranked by CDAls are mostly European cities, whereas those located in Africa and South America are rarely included. CDAls can not be mistakenly used to Europeanize the world, create a dominating Anglocentric view of sustainable development, or impose the implementation of strategies to achieve specific performance that may not meet the needs and expectations of local stakeholders.

j) Using sub-indicators to better capture the various facets of the principle: Capturing the various faces of the principles is the main objective of CDAls using composite indices. Determining the number of indicators remains an unsettled and evolving area. To avoid using a large number of indicators that aim to capture the various facets of an area of performance, the CDAI can be modified to include another level in its HSO. For instance, the EIU's Global Liveability Ranking uses four indicators under category 2 to capture the availability and quality of healthcare; these can become an indicator with four sub-indicators in order to include other meaningful indicators to better assess the healthcare system of a city or community.

k) Incorporating scientifically-based methodologies and processes to minimize subjectivity: Identifying, selecting, and assigning weight to elements (i.e., subprinciples, criteria, indicators) are not arbitrary activities and subjectivity should be minimized as much as possible. Instead of relying on a selected group of individuals (e.g., consultants), scientifically-based methodologies, processes, and approaches can be implemented. The area of operational research provides a number of techniques, tools, and instruments (e.g., decision analysis [e.g., multicriteria decision-making (MCDM)]) that can be incorporated into the composite index methodology. MCDM allows engaging the participation of shareholders and best capture their needs and expectations.

I) Trying to measure, compare, and rank the unmeasurable: As continuously evolving concepts, quality of living, liveability, and sustainability definitions are moving targets deeply embedded in subjectivity. To validate progress toward achieving performance improvement depends on the ability to accurately measure it; however, what cannot be defined faces obstacles or simply cannot be measured. While facets of sustainable development can be assessed with less subjectivity than others, identifying, selecting, and weighting the set of elements (i.e., sub-principles, criteria, indicators) within each level of the HSO that better capture a widely accepted holistic view of the concept has proven to be evasive.

m) CADls are not perfect, not one size fits all and should not be used as a stand-alone measure: CDAls are diverse and influential but far from perfect. Although CDAls provide easy access to compare cities and communities around the world based on a pre-determined set of elements (i.e., criteria, indicators), they should not be interpreted as an isolated or absolute standard of performance assessment. There are many CDAls, and not one size fits all. Each CDAI provide a specific idea of what sustainability, liveability, and quality of living should be for cities and communities. Stakeholders' visions and needs vary from city to city; therefore, the use of CDAls to compare the performance of cities and communities around the world with the same set of elements (e.g., criteria, indicators) may imply the standardization of the concept of sustainability and the notion of sustainable development. Consequently, CDAls and other assessment tools should be used as decisionmaking tools in conjunction with other assessment tools and data to evaluate strategies, plans, policies, or programs designed to improve sustainability, liveability, and quality of living in cities and communities.

In addition to the intrinsic challenges, shortcomings, and limitations in the development and application of composite indices to assess the quality of living, liveability, and sustainability performance of cities and communities, the use of indicators and composite indices face criticism along with skepticism and disadvantages [33], [34], [35], [36], [37]. Because CDAISs use composite indices as their assessment methodology, the research indicated that CDAls encounter similar challenges because of the lack of structured, transparent and widely accepted frameworks for the identification, selection, and weighting of the elements (i.e., sub-principle, criteria, indicators) included in each hierarchical level. The following list of observations supported by the findings of this study also contributes to developing areas of research and opportunities for improvement:

a) Relying on the developer or proponent's reputation to position the ranking - the reputation of the developer or proponent is an advantageous factor used to gain market recognition and influence the credibility of the CDAls.

b) Focusing on the audience's interests instead of focusing on the various facets of the principle-the CDAl focuses on the preferences of its audience (e.g., the EIU's Global Liveability Ranking and Mercer's Quality of Living Ranking give disproportionate weight to the preferences of highly-skilled professional and expatriates) instead of identifying, selecting and weighting elements in a way that better capture the essence of the principle.

c) Understanding indicators - users of CDAls are unaware of the meaning of indicators, the use of subindicators, or potential trade-offs implemented in the data aggregation process; therefore, the process of aggregating and weighting sub-indicators remains immersed in subjectivity and lacks transparency.

d) Limiting access to data-methodologies used by CDAls can only be adequately understood by analyzing the original data collected; developers or proponents of CDAls often provide limited details on identification, selection, weighting, and aggregation of elements within each hierarchical level.

e) Extrapolating data to fill data gaps - the data collection process may face collecting accurate data because the city does not collect data or regularly update it; CDAls need to extrapolate from partial data or national records. Similarly, data may be collected from the metropolitan area instead of the specific municipality.

f) Leaving data purposely left out or weighted deliberately - the processes of selecting and weighting elements (e.g., criteria, indicators) are designed to favor a specific group of participant cities.

g) Ignoring the CFSS - cities and communities around the world do not have the same view of the quality of living, liveability, and sustainability in part because their inhabitants have different needs and expectations; the CFSS better mirror the areas of performance that cities and communities are focusing on.

h) Focusing on a snapshot in time of the city's performance - CDAls not only provide a narrow view of the principle but also capture a snapshot of the performance of the city in certain areas of quality of living, liveability, and sustainability.

i) Standardization of terminology - users (practitioners and scientists) of FDC, CDAls, and other sustainability management and assessment-related information (e.g., journals, reports) find discrepancies and potential contradictions in terminology as illustrated in Table 2.

Challenges associated with the continuous state of theoretical evolution of the concepts has not prevented scientists and practitioners from developing and proposing processes, approaches, strategies, models, appraisals, and methodologies to assess, compare, and rank the quality of living, liveability, and sustainability of cities and communities. While scientifically based instruments and stakeholder and multidisciplinary decision-makers engagement and participation mitigate some of the procedural and methodological concerns, vagueness and ambiguity associated with lack of consensus on agreed-upon definitions and subjectivity embedded in existing proposed structured frameworks provide opportunities for improvement in the area of sustainability assessment studies [38], [39].

Due to the increasing popularity of CDAls using composite indices to assess, compare and rank performance, three main areas have gained relevance from the assessment methodology perspective: the HSO, the identification and selection of elements (i.e., sub-principles, criteria, indicators) in each hierarchical level, and the W&AS. Composite indices are practical but the perceived straightforwardness of the assessment methodology is often and misunderstood and underestimated. Assessing the effectiveness of CDAls in capturing the various facets of the principle (e.g., sustainability) is often focused on the number and type of elements within each hierarchical level (e.g., criteria, indicators); however, other factors have a determinant influence on the accurate application of the assessment methodology. Furthermore, the HSO supports each management process of the assessment methodology, and each of its hierarchical levels assists and supports the mission of the principle, whereas the W&AS determines the relevance and impact of elements within each hierarchical level and the overall value of the principle.

Drawing conclusions on the effectiveness of CDAls in capturing the various facets of sustainable development was assisted by FDC in the areas of identification and selection of elements within some levels of the HSO (i.e., criteria, indicators). Moreover, while the study included a detailed review of 3 representatives, credible and reputable CDAls, the analysis of the HSO and W&AS considered other CDAls in order to gain a better understanding of the application of composite indicators. As a result of a comprehensive research approach, observations and considerations were presented, 12 main issues were identified, which also represent areas for improvement and opportunities for future research, and eight areas of common skepticism and criticism were highlighted. Although there are intrinsic challenges, shortcomings, and limitations in their development and application, CDAls using composite indices have the characteristic of adaptability. In addition to the need for becoming more dynamic, CDAls can increase their effectiveness by engaging stakeholders and multi-disciplinary decision-makers, implementing scientifically based methods, aligning the identification and selection of elements (i.e., criteria, indicators) with FDC, and considering and incorporating the continuous evolution of the concepts.

Arguably, standardized assessment methodologies such as composite indices may not capture the needs and expectations of the cities and their inhabitants better than a customized urban development and sustainability plan. Therefore, effectively capturing the various facets of sustainable development not only depends on embedding the constant evolution of the concepts into the assessment process but also on identifying the stakeholders' vision of sustainability and meeting specific sustainability performance goals [28], [40]. The main argument surrounding the effectiveness is certainly outside the control of any CDAI; agreement around the very own concept of sustainability still has not been reached; therefore, unless sustainability is understood and the ultimate target is set, any ranking can argue its rightfulness and usefulness. As a result, whether or not CDAls capture the concept of sustainable development of cities and communities can be debated; the rightfulness of the approach implemented to capture the various facets of quality of living, liveability, and sustainability has been argued by each developer or proponent of CDAls using composite indices.

Highlighting the methodology to assess, compare and rank sustainability, liveability, and quality of living benefits developers, proponents, scientists, and mainstream users of CDAls. There are three main areas for improvement and research of particular interest to developers, proponents and scientists. A closer look at the terminology and HSO used by CDAls indicated the need for developing a standardlike harmonization in sustainability assessment in particular those methodologies using composite indicators. A second main area for future research can be found in the identification and selection of the elements in each level of the HSO. The essence of properly capturing the different facets of sustainable development relies on identifying and selecting the right number and type of elements (i.e., sub-categories, criteria, indicators, verifiers) that reflect the aimed vision. The W&AS provides another wide range of opportunities for improvement and research. Identifying and selecting the right set of elements must be complemented with the proper systems to weigh and aggregate the assessment outcomes.