Benefits and Challenges of Applying Data Analytics in Government Auditing

Digitisation has revolutionised completely how industries and sectors operate, leading to the present-day phenomenon of Data Analytics (DA). Collecting and storing voluminous data can be a burden for entities, but this burden can be turned into an opportunity through the use of DA (Russom, 2011).

In recent years, DA has been slowly infiltrating various areas, from manufacturing to medicine; and auditing is no exception. The International Auditing and Assurance Standards Board Data Analytics Working Group (IAASB DAWG) (2016, p. 7) provides the following definition for DA:

“…the science and art of discovering and analysing patterns, deviations and inconsistencies, and extracting other useful information in the data underlying or related to the subject matter of an audit through analysis, modelling and visualisation for the purpose of planning or performing the audit.”

The auditing profession is shifting from one which used to be standard and routine- based, to one which takes advantage of data in order to identify patterns (Van Schoten, 2016). The private sector is very often at the forefront of innovation, whereas the public sector tends to lag behind. Various Supreme Audit Institutions (SAIs) such as those of the United Kingdom (UK) and Australia are radically transforming public sector auditing thanks to the advent of DA, as indicated by INTOSAI (2018) and VAGO (2018) respectively. According to Lewis et al. (2014), embracing DA is a fundamental step forward if government auditing is to be modernised. It would allow the profession to add more value, both in terms of cost management as well as transparency and accountability (Lewis et al., 2014).

The National Audit Office (NAO) fulfils the role of SAI in Malta. The Maltese NAO conducts a variety of audits within the public sector, primarily financial and compliance audits, performance audits and special audits and investigations.

This study seeks to assess how DA can be implemented within the scope of Maltese external public sector audits in order to enhance the value obtained from these audits. More specifically, the objectives of the paper are to explore the role of DA within the NAO and to assess the applicability of DA in the various audits carried out by the NAO. The study aims to shed light on how SAIs can incorporate DA into their operations, in order to increase the efficiency, effectiveness as well as the level of insights gained from the audits they carry out.

The paper proceeds by presenting a literature review on the applicability of DA in the auditing profession. Due to the limited research on DA in the public sector, this paper also refers to the use of DA in the private sector. The research methodology is set out in Section 3. Section 4 will present the research findings while Section 5 concludes the paper by providing an analysis of the findings of this study.

DA in the different Stages of an Audit

According to Cangemi (2014, p. 1), "Data Analysis tools can be used to unlock hidden value and act as a key catalyst for delivering insights…" Traditionally, the use of DA in auditing revolved only around the execution phase; however, using DA throughout the entire audit is likely to reap the most benefits (Cangemi, 2014).

Planning

DA allows the auditor to identify high risk areas, and consequently to adjust the focus of the audit prior to the execution phase (Cangemi, 2014).

For instance, when auditing the accounts payable of an entity, if the relevant data is obtained during the planning stage, useful information can be extracted for the subsequent audit phases (Cangemi, 2014). The identification of the number of transactions undertaken during the financial year, as well as determining the company’s suppliers and purchasing patterns, allows for a deeper understanding of the business by asking specific, as opposed to generic questions (Cangemi, 2014). This initial analysis also guides the auditor to ask for the right documentation when conducting the fieldwork.

Execution

During this stage, DA can uncover exceptions, potentially identifying an issue which can then be used as audit evidence (Comptroller and Auditor General of India (C&AG India), 2017). Cangemi (2014, p. 3) argues that using advanced and specialised software is beneficial since it will produce more "tangible insights".

Reporting

Once an audit is concluded, DA can be helpful for presenting the findings to the intended users, allowing for a better understanding of those findings (C&AG India, 2017).

Types of DA

Primarily, there are four types of DA, which can be categorised according to their complexity and the value that they contribute, as depicted below.

Descriptive DA: What happened?

Descriptive analytics uses raw data from multiple sources to obtain meaningful insights about past events (Tschakert et al., 2016). However, such DA sheds light on a problem without providing any underlying explanation. Descriptive analytics yields descriptive statistics, such as, the mean, frequency, and distribution, as well as "vertical1 and horizontal2 analyses of financial statements" (Tschakert et al., 2016, p. 60). Data visualisation can be useful for presenting these statistics in graphical formats, thereby identifying patterns that may exist within such data (Certified Public Accountants of Canada (CPA Canada), 2017). According to CPA Canada (2017), there are various tools which can be used for data visualisation purposes, some of which include Tableau, ACL, Microsoft Excel and IDEA.

A survey carried out by CPA Canada (2017) revealed that approximately 46% of the external audit firms in the sample use DA, with 26% of these respondents using some sort of descriptive statistics.

Diagnostic DA: Why did it happen?

Descriptive analytics is a prerequisite to diagnostic analytics. While it is important to know what happened, it is not enough as one must also understand the cause (Tschakert et al., 2016). According to Vesset (2018a), diagnostic analytics has three main functions, namely, (i) identification of anomalies in the data; (ii) explanation of anomalies; and (iii) establishing relationships by looking into activities which gave rise to identified deviations.

Predictive DA: What is likely to happen?

The objective of such analytics is to make predictions using past data and analytical techniques (Edwards, 2019). However, the accuracy of such forecasting depends highly on the quality of the data as well as the situation's stability (Bekker, 2019).

Advanced statistical analysis is one type of predictive analytics. This involves the use of statistical techniques such as regression and cluster analysis, using such tools as ACL, IDEA and Python (CPA Canada, 2017).
According to the C&AG of India (2017), techniques like correlation and regression can be particularly useful for performance audits. This is because these audits seek to

1 Vertical analysis involves converting balance sheet figures to percentages based on total revenue (Maguire, 2018).

2 Horizontal analysis involves comparing financial data over a number of years (Maguire, 2018). assess whether any initiatives taken have been effective in achieving the required outcome, and whether this has been achieved in the most efficient way possible.

Prescriptive Analytics: What should be done?

As Figure 1 shows, this is the most complex form of DA, but also the one which adds the most value. It allows the auditor to prescribe an entire range of potential outcomes, depending on the inputs provided (Tschakert et al., 2016). The outcome will then help to support decision-making. This is referred to as optimisation, and can be performed using tools such as Excel Solver, Matlab and Gurobi.

Such analytics are best suited when the constraints are specific and highly accurate (Vesset, 2018b). However, this complexity requires advanced tools, which can make implementation and management an issue (Bekker, 2019).

Benefits

The following are the three principal benefits that can be gained from using DA:

Efficiency

An important benefit of DA is that it can be used to automate time-consuming tasks when conducting financial statement auditing (Cao, Chychyla, and Stewart, 2015). Auditors can be freed from "mundane tasks", consequently, freeing up more time to focus their attention where judgement is required (Manson, McCartney and Sherer, 2001, p. 120).

Quality

Auditors can use DA to test entire populations of data as opposed to selected samples (Earley, 2015). This is bound to generate audit evidence that is of higher quality as it allows auditors to consider a broader range of relevant data (Sirois and Shukarova Savovska, 2017). DA further puts auditors in a better position to identify anomalies which will inform risk-based decision making (Liddy, 2014). Anomalies are those instances where data deviates from an auditor's expectations, and thus, such occurrences must be investigated further (Earley, 2015).

Insights

Auditors provide insights on risk exposure, operations, and company performance, and can build an entire database of valuable information which can be used and developed year after year (Liddy, 2015). Increased use of DA during audits is facilitating the identification of anomalies and inefficiencies, providing valuable information for audit clients (Tysiac, 2017).

DA can also be utilised successfully to detect fraud (AICPA, 2014). For instance, Benford's law can be used to help identify illicit transactions (McGinty, 2014). This is a mathematical principle for predicting the frequency of digits across data, which can then be compared with actual data (Fay and Negangard, 2017). Categories of numbers which differ from set expectations may indicate an increased risk of fraud (Fay and Negangard, 2017), and thus, warrant further investigation.

Challenges

The advent of DA does not come without its challenges, with the following being some of the key issues:

Knowledge and Competencies

Traditionally, auditor training and education has neglected to consider factors such as the recognition of patterns and evaluation of anomalies; therefore, such skills can only be acquired after years of experience (Earley, 2015). Knowledge of IT is likely to become a very important aspect of the skills of any auditor (Vanbutsele, 2018).

Data Issues

There is the issue of accessibility of data, concerning the degree of access the auditor is granted, and the type of data he is granted access to (Earley, 2015). In fact, for the Massachusetts Office of the State Auditor (MOSA), access has indeed been a challenge, with a number of requests for data taking numerous months to complete (Bump, 2015). Additionally, even if the latter issues are resolved, the question of data integrity remains (Whitehouse, 2014).

Time and Cost

Auditors might end up wasting valuable time looking into identified anomalies which transpire to be regular business transactions (Whitehouse, 2014).

Implementing DA requires companies to make the necessary investments, particularly with respect to technologies. Indeed, in the research conducted by Dagilienė and Klovienė (2019, p. 764), one Big Four firm stated that "...by investing in analytical tools we always measure costs...as it's really very expensive".

Investment is also required in education. Another Big Four firm interviewed in the aforementioned research stated that, at one end of the spectrum, there are the IT specialists who know nothing about accounting, while at the other end, there are auditors with poor IT skills (Dagilienė and Klovienė, 2019).

Client Resistance

Joshi and Marthandan (2018) explain that data confidentiality may cause concerns for auditees, and could possibly contribute to client resistance if they are asked for larger amounts of data. Dai and Vasarhelyi (2016) propose appropriate policies aimed to keep data secure; one option could be using encryption and secure channels as a means of communication.

Risk of Litigation

CPA Canada (2017) found that the risk of litigation was the least popular challenge among participants, with this issue being selected by only 6% of respondents. Auditors should take the necessary measures to safeguard auditees' data so as to maintain confidentiality (Alles and Gray, 2016), or otherwise, risk exposing themselves to litigation.

The Activities of SAIs

According to Bump (2015), the approach adopted by the MOSA consisted of three phases. First, the Office aimed to build a "test environment" where DA techniques were developed, and internal procedures established. The second phase focused on consolidating what was learnt, investing in staff via training, and standardising the use of DA when carrying out tests and investigations. The final phase involved having fully trained auditors making use of DA, including the use of predictive analytics for projection purposes. Bump (2015, p. 54) explains that the introduction of "data analytics has changed how the entire staff utilizes critical thinking skills". Perhaps one of the most valuable contributions was that this Office improved the "policy impact of [their] end product" (Bump, 2015, p. 55).

Similarly, the Victorian Auditor General's Office (VAGO) has established a DA roadmap, divided into smaller initiatives which it plans to implement over a number of years, starting first with descriptive and diagnostic DA, and moving gradually to predictive DA (Jiang, 2019). VAGO also plans to train its staff in order to increase their skills and capabilities in using DA tools (VAGO, 2018).

The Queensland Audit Office (QAO) has been training its auditors, while making sure they have access to "data champions" for more complex issues (Close, 2016 cited in Rollins, 2016, para. 32). The DA team at the QAO initially placed its focus on implementing DA for performance audits (Williams, 2017). One such audit related to the performance of hospitals, where advanced statistical analysis was used to determine if setting a four-hour target rate for each patient admitted to an emergency department was effective (Close, 2016 cited in Rollins, 2016). This performance audit revealed that data was being manipulated to change the time of patient admissions. Consequently, the Office recommended that this target should be revised as it could have a negative impact on the patients' safety (Close, 2016 cited in Rollins, 2016).

In order to fulfil the research objectives, a mixed-methodology approach was considered the most appropriate. Mixed methods provide a better answer to the research question than would be possible with qualitative or quantitative methods on their own (Creswell and Plano Clark, 2011).

Qualitative Method – Interviews

The data collection instrument consisted of semi-structured interviews, comprising primarily open-ended questions. Three interview guides were designed for the different categories of interviewees, being NAO auditors, members of the DA Working Group within the NAO as well as higher level management. All interview guides were composed of three main sections, namely, (i) current use of DA; (ii) applicability of DA and (iii) the way forward; with each guide containing questions specifically tailored for the respective category of interviewee.

After the NAO was informed of the study, a number of auditors from the different units were contacted via email inviting them to participate in an interview. Table 1 below summarises the details of the interviewees and their respective section.

The interviews were conducted between December 2019 and February 2020, with each interview lasting between 50 and 90 minutes. All interviewees gave their permission to record the interviews, which were subsequently transcribed by the researcher.

Analysis of the qualitative data was made in accordance with Miles and Huberman’s (2014) approach which involves the condensation and display of data, followed by inference and verification of conclusions.

3 The DA Working Group members also hold the position of auditors within the Special Audits & Investigations Unit of the NAO.

Quantitative Method – Questionnaire

For the quantitative aspect of this study, a questionnaire was designed for distribution to NAO auditors. The questionnaire targeted auditors working in three different units, namely, Financial and Compliance (F&C) audits, Performance Audits, and Special Audits and Investigations. The majority of the questions were of a close- ended nature; however, some open-ended questions were included to allow respondents to further express their views.

The questionnaire was first reviewed by the Deputy Auditor General in order to obtain his views and approval. It was agreed with the NAO to distribute the questionnaire to three employees within the three different units being targeted by the questionnaire, for pilot testing purposes. Bell and Waters (2014) refer to pilot testing as a trial run for a questionnaire, adding that the aim of pilot testing includes assessing the clarity of questions, identifying the average time to complete, and the appropriateness of the questionnaire’s layout. In November 2019, a link to the questionnaire was sent to the three participants via email, explaining the purpose of the dissertation and inviting the respondents to comment on the various aspects identified by Bell and Waters (2014). Following some adjustments, the questionnaire was distributed by the NAO to all its audit staff via email, during December 2019. Table 2 below details the number of responses per section and the resulting response rates.

In order to analyse the data collected from the questionnaire, the researcher used the chi square test to test the independence between two variables (Saunders et al., 2016). One of these variables indicates the group (F&C Audit, Performance Audit and Special Audits & Investigations) while the other variable refers to a theme, such as the challenges or benefits of DA.

The null hypothesis states that no association exists between two categorical variables. This hypothesis is only accepted if the p-value is greater than the 0.05 significance level. The alternative hypothesis specifies that significant association exists between two categorical variables and is accepted if the p-value is less than 0.05.

This section presents the empirical evidence gathered from the semi-structured interviews and the questionnaires. The findings delve into the existing level of use of DA within the Maltese NAO, as well as the potential applicability of DA in the different types of audits carried out.

Current Use of DA

Both interviewees and questionnaire respondents were asked whether they utilise any DA techniques during the audits.

Descriptive & Diagnostic Analytics

Table 3 illustrates that only 5/16 of questionnaire respondents within the F&C unit make use of these analytics when performing their audits. The relative usage of such analytics is slightly higher in performance audits, and even higher in the Special Audit and Investigations department (4/9 and 4/6,respectively).

Questionnaire respondents were asked to select the different stages of an audit in which this type of DA is being used (see Table 4).

An interviewee from the F&C section explained that during the planning stage, DA has been used to select a sample of direct orders which required investigation. The list of all direct orders was reviewed, and each direct order was assigned a risk rating, based on the following four criteria: the direct order value; the number of direct orders awarded to the same supplier; the number of direct orders awarded in the same month; and an indicator of risk based on the nature of the direct order. A sample of 10 direct orders from the population was chosen for further analysis based on this risk rating.

Another interviewee from F&C mentioned using descriptive DA during the audit reporting stage, when looking into direct orders and negotiated procedures. A number of charts were extracted from the data, illustrating the situation within the Direct Orders section.

An interviewee from the Special Audit and Investigations unit explained that descriptive DA was used in an audit aimed at examining the allocation of funding to local councils by central government. The interviewee further explained how another audit within this unit sought to investigate from which regions the applications for Algerian visas were originating. The application and acceptance rates per province were plotted on a map to highlight those provinces with higher application and acceptance rates. The interviewee explained that with such geographical representation, it became clear that regions which are closer to where business activity is concentrated were those with higher rates of acceptance.

Predictive Analytics

Participants were asked if they make use of predictive analytics, such as, advanced statistical analysis. None of the questionnaire respondents or interviewees within the F&C unit have ever made use of this type of DA. Moreover, only one respondent (1/9) from the Performance Audit section and one respondent (1/6) from Special Audits have used this DA in their work. When compared to descriptive and diagnostic DA, the tendency for NAO auditors to make use of predictive DA is much lower.

The interviewee within the Performance Audit unit explained how predictive analytics was used in a performance audit aimed at assessing the efficiency and effectiveness of the Occupational Health and Safety Authority. Regression was applied to look at various quantifiable factors in order to explain accident rates in the construction industry. The explanatory factors considered by the NAO included hours worked, age of workers, and weather conditions. The analysis enabled the Office to highlight the fact that high temperatures are the only factor which tend to be highly correlated with accidents recorded within this industry.

Within the Special Audit and Investigations unit, correlation was used when investigating Electrogas, the entity which operates and manages the power plant in Malta. The Office was required to analyse whether the combination of volumes of power purchased from different sources was economical. This involved the comparison of electricity purchased at a fixed rate from a new power plant with electricity purchased at a variable rate from the interconnector.

Prescriptive Analytics

Questionnaire respondents and interviewees were also asked if they make use of prescriptive analytics, such as, optimisation. None of the participants from any of the different units have ever made use of this type of DA in their work.

Questionnaire respondents and interviewees were asked whether they believe that the various types of DA discussed above are applicable to their work. Only those questionnaire respondents who are not using DA were asked about such applicability.

F&C Unit

The greater majority of questionnaire respondents in the F&C unit who do not make use of descriptive and diagnostic DA (10/11) believe that their work can benefit from such DA. One respondent from the F&C unit claimed that:

“We will be able to collect more significant information from large volumes of data, hence better understand ministries/departments and other entities being audited. By utilising information obtained, our audits will provide more coverage and better assurance.”

Once again, the majority of respondents believe that their work can benefit from the use of predictive DA (15/16). However, some respondents explained that although they believe it may be helpful, they are unsure how it can be used (8/16). Other respondents commented that it can provide more comfort to their audits and more relevant information. With respect to prescriptive analytics, only half of the respondents believe that this type of DA can be useful (8/16).

Interviewees within the F&C unit (2/4) explained that, to date, the financial statements of ministries and departments have been prepared on a cash basis. This creates difficulties for NAO auditors since information is not integrated, with some accrual data being available only on Microsoft Excel or on paper. Although the process of adopting accrual accounting has commenced, the NAO has no visibility as to how data will be generated by auditees, which will in turn impact the implementation of DA.

Performance Audits

The majority of questionnaire respondents from Performance Audits who do not currently use DA (4/5) argued that descriptive and diagnostic DA would help to reduce the need for sampling, and provide a much more complete and precise analysis of data.

With respect to predictive analytics, a large number of questionnaire respondents believe that this type of DA is applicable to their work (7/8). According to one respondent:

“DA can broaden the scope of audits as it offers a faster means to analyse various datasets; it can also increase the audit team's options vis-a-vis topic selection, particularly in subjects that were previously considered as overly- complex.”

The percentage of respondents who deem prescriptive analytics to be applicable to their work turned out to be the lowest (6/9). These respondents were unable to provide practical examples of how this DA is applicable to their work.

The interviewee from this section (1/1) stated that the NAO must first master the use of descriptive and diagnostic DA before other types of DA are considered. This auditor believes that:

“With performance audits, we are assessing if Government should have done this rather than that…it is not as simple as seeing a set of financial statements.”

Management (1/1) argued that the use of DA might have resulted in more valuable insight for some of the performance audits conducted. For instance, while undertaking a performance audit on public transport, auditors could have requested data on the departure times of buses to determine how late buses tend to be. When management enquired with the auditors why this data was not analysed, the reasons provided were either because it takes time, or the auditors did not know how.

Special Audits & Investigations

All questionnaire respondents who do not currently use DA (2/2) and interviewees (2/2) agreed that descriptive and diagnostic DA are applicable to their work. In fact, both interviewees explained that data visualisation (descriptive DA) is often used for reporting purposes.

However, both interviewees explained that not all audits necessarily require an analysis of data. Where the nature of the investigation looks at a contract, one needs to read a contract in detail, and here, there is no purpose for DA.

Most questionnaire respondents deem predictive DA to be applicable to their work (4/5). This number is slightly lower however with respect to prescriptive analytics (3/6). The interviewees from this unit (2/2) do not see much scope for advanced analytics such as regressions for the purpose of investigations. One interviewee emphasised that such DA is very useful when one is looking at trends and patterns; however, when it comes to investigations:

“I am not sure if prescriptive and predictive analytics fit within our line of work. Part of the audit report is to find shortcomings and draw conclusions on them, and then provide recommendations on how things can be done better.”

Questionnaire respondents and interviewees from the different units were asked about the benefits obtained from the use of DA (see Table 6).

Efficiency

This advantage is mostly favoured within the F&C unit (6/16). A number of auditors from Performance and Special Audits are also of the opinion that efficiency is a valuable benefit of DA (2/9 and 2/6, respectively). All interviewees from the F&C unit (4/4) claimed that DA saves significant time when it comes to reconciliations, with one interviewee explaining that:

“DA can help with the acceleration of the process. There are certain tasks that can be very tedious and repetitive. If you usually opt for a sample of say 5%, with the appropriate tools and expertise, you may be able to consider the entire population within the same period of time.”

Quality

This benefit is mostly favoured by respondents from the F&C unit (5/16), followed by respondents from Performance Audits (2/9). None of the respondents working in Special Audits and Investigations selected quality as a benefit of DA.

Management (1/1) opined that the use of populations rather than samples enables auditors to be spot on, removes margin of error, and provides higher confidence levels.

All interviewees within the F&C unit (4/4) and Special Audits and Investigations (2/2) are of the opinion that DA is beneficial in the identification of outliers, which then warrant further investigation. Management (1/1) explained that:

“We need datasets which enable us to see what is financially material…if the

data reveals figures which are not up to your expectations, then you investigate.”

Insights

The ability to obtain more insight and understanding was a popular selection within the Special Audits and Performance Audits units (4/6 and 5/9, respectively). According to an interviewee from the Special Audits and Investigations (1/2):

“Insights are important. You do not get the same level of insight from a sample. To be able to say with precision that there is a leak of tax of X million, and you are certain of this because you have looked at all data, is very powerful; insight would trump all others in my opinion.”

The interviewee from the Performance Audit unit argued that DA can provide insight by evaluating larger and different data sets in combination.

Questionnaire respondents were provided with a list of challenges which may be inhibiting the further implementation of DA, and were asked to select those challenges which they deem to be relevant (see Table 7).

Knowledge and Competencies

Inadequate knowledge for effectively using DA is a popular challenge among questionnaire respondents, across all units. Interviewees mentioned that this is especially the case for prescriptive and predictive DA. The interviewee from the Special Audit and Investigations unit believes that investing in the capabilities of staff is a must, particularly providing training on different tools.

The questionnaire respondents who participated in training on DA (7/31) opined that the training provided to date is not sufficient. A respondent believes that training on the available methodologies is a must, while another proposed training on the different software available on the market. All interviewees (8/8), including members of the DA Working Group, believe that the way forward for applying DA effectively is via training. In addition, both DA working group members interviewed explained that, to date, there are no formal internal procedures on how auditors should utilise DA.

The interview held with a management member revealed that the Office plans to invest in its DA capabilities. In this regard, the Office aims to collaborate with and learn from what other SAIs are doing. Moreover, the Office is also keeping in mind DA related skills when recruiting. However, according to management (1/1), NAO does not have a DA strategy in place.

Data Issues

Issues with the data provided is another popular challenge, particularly among questionnaire respondents from the F&C unit (46.2%). An interviewee within this section (1/4) referred to the data issues encountered whilst performing an audit of Mater Dei Hospital's income. The system used to collect revenue is not linked to the other systems of the hospital. For instance, if one looks up a patient on the system who is not entitled to free health care, one would have to go through different databases to identify the services this patient received for billing purposes. The interviewee also mentioned that in some instances data was only kept on paper.

Therefore, the effective use of DA depends on what data an entity keeps and how that data is kept. All interviewees from the F&C unit (4/4) argued that if data is not captured, then that is a non-starter. This was also highlighted by the performance auditor, who claimed that in the case of social audits, data is rarely available. This auditor argued that:

“Data problems are a challenge throughout the entire NAO. It either does not come, or it is unstructured and poses great challenges with respect to its reliability.”

Time and Cost

The issue of time was selected by five respondents from the Performance Audit unit (27.8%), and was also mentioned by the interviewee from this section. According to the interviewee, given that an audit typically needs to be concluded within six to eight months, the time available for auditors to experiment with DA is limited.

The ‘cost’ issue was highlighted by all units, particularly the F&C unit (7.7%).

However, none of the interviewed auditors mentioned this limitation.

Client Resistance

This issue was selected by one questionnaire respondent from the Performance Audit section. Some of the auditors interviewed (3/7) and management (1/1) further explained that GDPR is often used as an excuse in order not to provide the data. Management continued that these concerns are unwarranted since GDPR only relates to personal data, and in such cases, data can be pseudonymised.

Risk of Litigation

This risk was only selected by one respondent from the Performance Audit section. Most of the interviewees clarified (5/7) that the Office entered into memoranda of understanding (MOUs) with auditees which explained how data is safeguarded and who has the right to access it.

The empirical evidence gathered indicates that while the NAO has started to integrate DA in its operations, the use of DA by the Office is still quite limited. The data gathered also suggests that DA will be beneficial to auditors working in the different units, as discussed further below.

Descriptive & Diagnostic Analytics

Whilst being described as being the most commonly used and simplistic form of DA, its usage at NAO is rather on the low side, with only 41.9% (13/31) of questionnaire respondents utilising such DA. Given its simplicity, this category of DA would be a good starting point; in fact, initial implementation of DA by the VAGO focused specifically on this type of DA (Jiang, 2019).

The use of DA for selecting a representative sample selection as suggested by an interviewee from the F&C section is in line with Cangemi's (2014) argument that during the planning stage, DA can be very useful to identify higher risk areas.

Similarly, Vesset's (2018a) three-stage approach to diagnostic DA is in line with the approach adopted to conduct the Algerian visas audit. This approach involved the identification of anomalies via the graphical representation of application and acceptance rates, discovering possible explanations and subsequently establishing relationships which can help explain such occurrences.

Predictive Analytics

The findings of CPA Canada (2017) are also manifested in the Maltese scenario where only 6.5% of questionnaire respondents make use of predictive DA.

It is also noted that from the examples provided by the interviewees, the single technique that was used in this regard was regression analysis, which further reveals that the application of such DA is quite limited. Edwards (2019) refers to additional forms of statistical analysis and to machine learning, as other types of predictive DA that can be applied (which the interviewees/questionnaire respondents are not utilising at NAO).

Prescriptive Analytics

The lack of use of such analytics may be due to the complexity associated with such DA. This also includes the issues associated with management and implementation of DA (Bekker, 2019). Moreover, it might be ideal for any Office to first establish and master the use of the simpler forms of DA, prior to implementing prescriptive analytics. In fact, this was the approach upon which the DA roadmap of the VAGO was built (Jiang, 2019).

F&C Unit

When it comes to SAIs implementing DA, F&C is the area which is often considered at first, as was done by VAGO (Jiang, 2019). The findings indicate that DA is being implemented in this area within the local NAO; however a significant number of auditors are still not applying DA in their work.

Financial audits are likely to lend themselves well to DA due to the inherent repetitiveness that is often associated with these audits, in line with Williams (2017). This was also reflected upon by an interviewee from the F&C unit (1/4), who referred to those tasks which are “tedious and repetitive” as being ideal for the application of DA.

Performance Audits

Unlike financial audits, which tend to be of a repetitive nature, performance audits are often unique, and one would need to be creative on how to best go about these audits. This is evidenced by the different examples of performance audits presented in the literature which have been carried out by different SAIs, such as VAGO (2018) and QAO (Close, 2016 cited in Rollins, 2016; Williams, 2017).

The interviewee within this unit argued that predictive DA may be challenging in performance auditing since the purpose of these audits is to assess if Government’s actions have been effective. However, C&AG India (2017) claims that this type of DA lends itself well to performance auditing because one can make use of correlation and other similar techniques in order to assess the effect of government’s intervention on the outcomes achieved thereby assessing the effectiveness of those actions.

Special Audits & Investigations

Literature is very scant on this particular area. SAIs which are implementing DA tend to focus on the other two areas discussed above. The NAO still applies DA within this unit, even though its use may not be as prevalent as it is in the other units. The use of data visualisation (descriptive DA) for reporting purposes is useful, and it has been implemented by the NAO in previous audits, as in the Algerian visas audit.

Efficiency

The benefit of being able to reduce the amount of time required to conduct a particular task is particularly identified by auditors within the F&C unit, although it was also noted by auditors from the other units. This is in line with the literature (Cao et al., 2015; Manson et al., 2001).

Quality

DA enables auditors to consider an entire population of data rather than looking at a sample (Earley, 2015). This is particularly true for financial auditors who deal with significant amounts of data. The interviewees’ comments in this regard relate to the fact that this would help to reduce, if not, eliminate the risk of not identifying errors. Moreover, the auditors’ ability to identify and focus their attention on where it matters, as explained by management, is in line with Liddy (2014) and Earley (2015), who argue that DA enables auditors to delve into identified anomalies.

Insights

In line with Liddy (2015), an interviewee from the Special Audit and Investigations unit highlighted the usefulness of DA for accumulating data over a number of years to establish patterns so as to help inform which department should be audited. An interviewee from the Performance Audit unit referred to the fact that one can obtain valuable insights by being able to evaluate different datasets together and identify discrepancies between the datasets.

Inadequate Knowledge

Traditional education for auditors does not consider the advent of DA, and this can only be compensated for through training and eventually experience (Earley, 2015).

The commitment to invest in the skills of auditors via training can also be noted throughout various SAIs, such as MOSA (Bump, 2015) and VAGO (Jiang, 2019).

DA requires the skills of different people with different backgrounds. It would be ideal to have someone who is both an auditor and a DA expert. In her dissertation, Vanbutsele (2018, p. 35) highlights that the ideal auditor would be the one who masters three different careers, namely:

“…a civil engineer, who possesses analytical skills; a business engineer, who possesses IT skills; and a student [of] business economics, who possesses business and economical skills.”

Data Issues

This issue of data integrity as highlighted by Whitehouse (2014) transpired to be a key matter across all units. The problem of accessing data was also mentioned by interviewees, which is reminiscent of Earley (2015) and MOSA, who found that certain requests for data took months to fulfil (Bump, 2015). For the local NAO, certain data which they requested from auditees never arrived.

Time and Cost

The interviewee from the Performance Audit unit mentioned that time restrictions hinder auditors from experimenting with DA. This issue was also selected by questionnaire respondents from this unit (27.8%). A possible reason why auditors may be reluctant to use DA is that the NAO has started imposing timelines for audits, so this could be disincentivising auditors from trying out new techniques in order not to go over the allocated timeframe.

Significant costs are often necessary, both in relation to IT investment, including software, and investment in the skills of staff. However, one can reduce the costs involved by opting to start with off-the-shelf software which would be cheaper than opting for customised software. Moreover, some software, such as, Python and R, are open source, and can thus be utilised free of charge.

Client Resistance

The issue of data confidentiality mentioned by participants is in line with Joshi and Marthandan's (2018) arguments. Thus, it is imperative that the Office explains to its auditees the purpose of the requested data in order to reassure them.

Risk of Litigation

This issue was not very popular with auditors. This is most likely because the Office entered into MOUs with auditees in order to safeguard against such possibilities. Notwithstanding, Alles and Gray (2016) refer to the importance of safeguarding data security since breaches can give rise to litigation.

The study suggests that all units across the NAO stand to benefit from the implementation of DA. In order for the NAO to take advantage of DA, the Office should fully commit to making the necessary investments to prepare for the future of auditing.

The NAO should devise a DA strategy which maps out the direction the Office intends to take in this regard, as has been done by VAGO (Jiang, 2019). The strategy should address the investment requirement in IT infrastructure to cater for larger amounts of data, while ensuring the security of that data. It is also recommended that this strategy incorporates what should be achieved in the immediate, medium and long term.

As has been highlighted earlier, training is key for addressing any lack of expertise in this area (VAGO, 2018). Auditors should understand how DA can be of assistance in their work. Whereas it is important to provide advanced training to the DA Working Group, basic training is imperative for all auditors within the NAO. Consequently, all auditors would be able to carry out basic DA techniques independently, especially for financial audits, and then refer to the more knowledgeable DA working group for more complex tasks.

The nature of the work carried out by public sector auditors all over the world is particularly distinguished by how the public sector in each respective country is organised. This means that what may be applicable to one SAI may not be to another, and it is thus up to each country to re-invent itself in this area in a manner that is beneficial to its circumstances. As former Auditor General of QAO Anthony Close claims about his Office’s journey towards DA, it is “not for the faint- hearted…[because]…[t]he rulebook has not been written in this space” (cited in Rollins, 2016, para. 24).