Heuristic Evaluation of E-Learning

As Knowledge Management applications, E-learning platforms have been used in many organizations. Universities as knowledge management appliers and early adopters of Elearning platforms as knowledge-sharing channels, making education independent of time and location, created new opportunities for students to become active and collaborative participants of their learning. In this respect, learning management systems offered tools and modules to facilitate knowledge construction, knowledge gathering and sharing among its participants. This study examined the critical factors of usability evaluation of the learning management system Moodle as part of students’ blended learning in a computer literacy course at a funded University in Turkey. The model based on Nielsen’s 10 heuristics and was tested with data from 236 students. Descriptive statistics showed that students generally agreed with the system and did not face problems when working with it. The study used structural equation modeling (SEM) for data analysis and found eight factors to be significant in the research model after an exploratory factor analysis.

mechanisms as knowledge-sharing channels.Employees can be able to find out what they need so that they can apply their expertise to the organization.Reuse of knowledge not only increase efficiency and effectiveness but also free up workers to devote themselves to create innovative knowledge to be added to corporate memory.As examples of reusable knowledge, learning objects such as a learning module on a given topic, lecture slides, a test, a demonstration, or combinations of different content formats including multimedia, continue along the KM cycle as they shared, disseminated, and applied by other users (Dalkir, 2011).
Universities as knowledge management appliers and early adopters of E-learning platforms as knowledge-sharing channels, making education independent of time and location, created new opportunities for students to become active and collaborative participants of their learning.Teachers also gain many opportunities by creating digital materials and use them in their online courses.Teaching and learning passed beyond the restricted traditional classrooms.With the delivery of digital course contents, management of the online course and tracking the learner progress, learning management systems provided a solid technology-enhanced learning environment.As a learning management system (LMS), Moodle allowed students to follow their online class via internet asynchronously and to have the opportunity to chat with their peers and instructors via forum synchronously as well as to download the digital class materials and submit their homework.
However, the complexity of such systems having many features to be used by employees, students and instructors, also raised the questions of how users of these system can use them effectively.There have been many studies which analyzed these systems for general usability testing purposes from pedagogical and institutional perspectives.Some of them were on Moodle LMSs.
They generally tried to find out the usability problems in a qualitative manner instead of overcoming and improving them, and retest the LMS.Focused on the usability analysis perspective of E-learning systems, Squires and Preece (1999) reviewed the existing usability checklists and criticized them in a sense that most of the usability heuristics-based checklists lacked consideration of learning.They proposed a usability heuristics-based on an extension of Molich and Nielsen's (1990) predictive evaluation methodology.
In this respect, this study evaluates the usability of Moodle E-learning platform in a funded university in Ankara.In the following sections, first the theoretical background and hypotheses are indicated.Then, the focus is on the methods that are used and on the usability analysis results that are presented.The article is ended with conclusions drawn and discussions how to create a usable Moodle LMS as a Knowledge Management E-learning platform.

Theoretical Background
According to the international standard ISO, the definition of usability is "The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user."Nielsen (1994) spoke about the concept of web usability and stated that web pages that are simple and organized help the users in finding the information they are searching for.He defined the usability as the combination of factors that affects the interaction of the user and the product or the system.
There are three evaluation methods in assessing the usability; methods based on the inquiry, methods based on the investigation (heuristic evaluation) and user tests.Methods based on the inquiry include the data collection methods that can be conducted with questionnaire, interview and focus groups.Methods based on the investigation (heuristic evaluation) are the methods that the system would be evaluated by the experts with the usability criteria especially in the design and production steps.In this evaluation method, the most knowns are 10 usability heuristics, Shneiderman 8 golden rules ve Norman's 7 laws of interface (Çağıltay, 2011).If there is a low experience then the evaluation would be with design manuals, usability criteria, ergonomical principles and standards.The reliability would increase if the evaluation could be conducted with more than one expert.User tests are the tests conducted before the product emerge in the market.The users test the software (a web site, game or educational software), at the same time their eye movements would be tracked and the environment would be registered with a camera.Many tasks required completed by the users, where the users focus and how long they would spend time, would be identified with an eye-tracker device.Nielsen (1994) explains the concept of usability with the measurable usability parameters such as measurable usability criteria since it cannot be measured in fact.These criteria reflect the users' thoughts on the system.Nielsen (1994) also pointed out the importance of the reliability and validity issues in user tests.Reliability is if the same results occur when the test repeated whereas validity is if the obtained results would reflect the required results.Some of the prior usability studies were as follows: A study by Çelik (n.d) proposed a set of criteria in the evaluation process of websites in terms of usability and content qualities and found that the most important factor was the interaction between the user and the management of the website.Evcil and İslim (2012) investigated usability with five factors such as effectiveness, efficiency, satisfaction, learnability, memorability.
To evaluate the usability of "Web Macerası", a study of integrating educational technology into curriculum by Gülbahar, Kalelioğlu, and Madran (2008) showed that both students and teachers have finished the process with success of 75% and informing the users, using different concepts for some processes and providing extra links to some procedures would increase the usability of the system.Bayrak, Karaman, & Kurşun (2014) determined the usability problems in the perspectives of effectiveness, efficiency and satisfaction and found out the main problems were on hardware and software inabilities to meet teachers' expectations in their study of interactive LCD panel whiteboards used in the Fatih Project.Kakasevski, Mihajlov, Arsenovski, and Chungurski (2008) on the evaluation for web applications balanced the heuristic evaluation, questionnaires and task-driven techniques.To make improvements during the iterative process, heuristic evaluation usually was to find the usability problems early in the design course.With the usage of questionnaires and the taskdriven techniques (multiple choice, scaled-answer and open-ended questions, and different task to do), the users evaluated the usability of the learning management system Moodle.
In Uçak and Çakmak's (2009) study of the usability of a web page of Department of Information Management in Hacettepe University, qualitative and quantitative methods were used; first stage was giving a pre-test for determining users' general computer skills, second stage was implementing a classical usability study of fourteen questions with the usage of the web page, third stage was giving a post-test to get the ideas of the users about the web page.The negative usability factors were difficulties about the content design of some parts and the positive usability factors were speed and the easiness of the navigation of the web page.=Dalcı, Alçam, Saatçioğlu, and Erdal (2008) explained new adjustments in the redesign of METU library webpage by determining the negative sides of it to be used more efficiently and effectively by the users.They conducted an usability test to the selected eight users and found that new library web interface was used more effectively and efficiently by the users.Çakmak, Güneş, Çiftçi, and Üstündağ (2011) developed a web site usability scale with a construct of four factors including 25 items and applied on 239 e-learners to determine the usability level of the LMS web site and results were reported.The scale was aimed to determine the usability level based on user perception and was considered to compensate the need of valid and reliable scale.Rusu, Munoz, Roncagliolo, Rudloff, Rusu, and Fifueroa (2011) focused on virtual worlds in their study and realized the need for the usability evaluation methods.A set of 16 specific usability heuristics and 49 items usability checklist were developed.The set of 16 usability heuristics were grouped in three categories: (1) Design and Aesthetics, (2) Control and Navigation, (3) Errors and Help.A group of evaluators (usually from three to five) inspected the interface design based on a set of usability heuristics and had their findings aggregated in a single list of usability problems.
The study of Martin, Martinez, Revilla, Aguilar, Santos, and Boticario (2008) presented a heuristics usability evaluation of three eLearning platforms comparing Moodle, Sakai and dotLRN.Five usability experts filled a data log sheet which was based on the 10 Nielsen Heuristics, which are detailed into more than 300 usability checkpoints following a task-based approach.The results showed that dotLRN was in the first place whereas Sakai second and Moodle third.
In another study, Çağıltay, Çiçek, Karasu, Çakır, and Akıllı Kaplan (2014) intended to develop a project OZTEK in which an innovative technology enhanced learning environment supported the education of students with special needs and to present usability issues by investigating effectiveness of such learning environments.Delice and Gungor (2009) presented a study aimed at detailing the usability problems of web sites with heuristic evaluation and analytic hierarchy process.The most important problem was found the consistency criterion.The second most important usability problems of the web site were found as documentation and error prevention.Elçiçek and Bahçeci (2016) investigated the usefulness of a Moodle-based system designed for "Teaching Practice Course" and the results showed that most of the users found the system useful and Moodle increased the efficiency of the course.Yılmaz and Tufekci (2013) evaluated the usability level of a Mathematics software interface and found successful when examining the users' behaviour in completing the tasks.
Sutcliffe and Alrayes (2012) indicated that Blackboard was perceived to be more usable whereas SecondLife provided a better user experience in their studies on collaborative learning in Second Life.Worst performance was indicated by dislike of avatar in Second Life and poor experience in Blackboard whereas better performance was associated with engagement with avatars and better usability in Blackboard.Machado and Tao (2007) compared the usability and effectiveness of two competing learning management systems; Moodle and Blackboard.They concluded that Moodle learning management system is the more efficacious and effective learning management system than the Blackboard learning management system since its course material organization and communication functionality was rated higher.
Oztekin, Kong, and Uysal (2010) proposed a quality and usability checklist, UseLearn for eLearning systems.In their study, they used structural equation modeling to validate that checklist quantitatively.The results of the study confirmed that UseLearn with its quantitative methodology provides an effective guidance for usability analysts in the design stage of eLearning systems.Oztekin, Delen, Turkyilmaz, and Zaim (2013) proposed a new machine learning-based evaluation method for assessing the usability of eLearning systems in their study and found that the proposed methodology helps to identify the usability factors and the eLearning system problems.Ssemugabi (2009) in his comparative study, heuristic evaluation by experts and survey evaluation among the learners were conducted and the findings showed that there was a high correspondence between the results of the two evaluations and heuristic evaluation was found an easy to conduct, appropriate, effective and sufficient evaluation method to identify usability problems.
Heuristics are some general principles that are used in usability evaluation of any computer interface.Nielsen (1994) stated that the result of any heuristic evaluation is a list of usability problems in the system, with reference to the set of heuristics used.If an eLearning system has usability problems, the reason should be identified and solved.This research discussed on the usability of Moodle by applying the heuristic evaluation to identify the problems students face when using a LMS.Therefore, a heuristic evaluation was conducted with five experts and using feedback, Nielsen's 10 heuristics were gathered into checkpoints in a questionnaire to get the students' opinions.By identifying those problems, this study also calls for a systematic methodology so that limited resources such as time and money can be used efficiently and effectively.
The research question is: What are the critical factors which influence e-learning according to the heuristic evaluation of the usability of Moodle LMS?
The following sub-questions will also be answered; a) How does the visibility of the system affect the status?b) In what way does the system and day to day life match one another?c) How does the control of the system impact upon the freedom of use? d) How does the consistency of the scheme about standards adhere to the compliance?e) How does the system prevent errors occuring through effective usability?f) In what way does the system guide through each step by recognized commands?g) How much flexibility is in the system to provide efficient usage?h) In what way does the design assistance user through minimalistic appearance?i) How do the error messages clearly indicate to the user what the issues are?j) When the help facilities are available, how clear and concise and how much assistance do they provide?

Methodology Heuristic Evaluation
A project OZTEK heuristics were gathered into checkpoints in a questionnaire to get the students' opinions.

Research Model and Hypotheses
This study used the model based on Nielsen's 10 heuristics (Figure 1).Those 10 Nielsen Heuristics were as follows: 1. Visibility of the system status: Users should know where they are within the system.H 1 : Visibility is positively associated with Usability.2. Match between the system and the real world: The metaphors used should correspond to real world objects and concepts.H 2 : Match between the system and the real world is positively associated with Usability.3. User control and freedom: Users should be able to exit the system at any time when they need to do so.H 3 : User control and freedom is positively associated with Usability.4. Consistency and standards: The system should be consistent in that the same words, situations, or actions refer to the same thing.H 4 : Consistency and standards is positively associated with Usability.5. Error prevention: Apart from giving good error messages, the system should be designed to prevent errors from occurring.H 5 : Error prevention is positively associated with Usability.6. Recognition rather than recall: Objects, actions and options should be visible, so that the user does not need to recall information.H 6 : Recognition is positively associated with Usability.7. Flexibility and efficiency of use: The system should facilititate the use for the efficiency of the users whether they are novices or experts.H 7 : Flexibility and efficiency of use is positively associated with Usability.8. Aesthetic and minimalist design: System should not contain irrelavant information since it may diminish the visibility.H 8 : Aesthetic and minimalist design is positively associated with Usability.9. Help users recognise, diagnose, and recover from errors: The system should give error messages which should indicate precisely what the problem is and suggest constructive solutions.H 9 : Help users recognise, diagnose, and recover from errors is positively associated with Usability.
10. Help and documentation: The information provided should be easy to search, be focused on the user's task, and should list concrete steps to be carried out by the user of the system.H 10 : Help and documentation is positively associated with Usability.
The independent variables of the model are Visibility of the system status, Match between the system and the real World, User control and freedom, Consistency and standards, Error prevention, Recognition rather than recall, Flexibility and efficiency of use, Aesthetic and minimalist design, Help users recognise, diagnose, and recover from errors, Help and documentation.The dependent variable of the model is Usability of the eLearning platform Moodle.

Research Setting
The data were collected at a major university located in the capital of Turkey.At this university, the usage of LMS was promoted as a part of the computer literacy course as well as in other courses.The course was offered in 48 sections to all of the students excluding engineering students at the university during the 2014-2015 Academic Calendar Year.Lecturers were encouraged to adopt the LMS Moodle in their teaching while students used the eLearning system for retrieving teaching materials and get informed about the class-related topics.Moodle was used as an eLearning platform in which instructors uploaded their teaching material, course syllabus, and all other class-related topics and managed the lectures.Students used Moodle to download the material, follow the schedule, join class discussion and interact with the instructors.

Participants
At the end of the spring semester of 2015, out of the 774 students in 48 sections enrolled in the Computer Literacy course at a funded University, 236 students answered the online survey questionnaire (Google Docs), resulting in a response rate of 30% of those using Moodle as a part of their course.The response rate was not high even though volunteer students were rewarded with 5% bonus points in their overall course grade compared to 58% of response rate of the eLearning satisfaction questionnaire.Table 1 summarizes the descriptive statistics while Table 2 gives their computer skills levels.There were more women than men since only one third of the 236 students were men (167 men versus 69 women).All respondents were undergraduate students, and their ages ranged from 18 to 24.Demographics and background of survey participants are indicated in Table 1 and Table 2.

Instruments
The instruments were adapted from original language based on previous literature (Nielsen, 1994) by consulting to an expert for language translation.A 5-point Likert scale questionnaire ranging from 1 as strongly disagree to 5 as strongly agree was used for the measurement.The scale has been studied and its content and structure have been reviewed by a colleague.Questionnaire was delivered after using Moodle one semester.Table 3 summarizes the measurement constructs and codes used in this study.

Pilot Test
An online pilot test was conducted with a class of 20 computer literacy students who used the LMS Moodle as a part of their course.According to the results of the pilot test there was no change in scale items.The pilot test results were excluded from the actual study.

Data Analysis
This study used structural equation modeling (SEM) implemented for data analysis.SEM analysis was chosen over regression analysis because SEM can be used to analyze all the paths in one analysis.First, exploratory factor analysis was used to develop the measurement model.Second, to explore the causal relationships among all constructs, the structural model for the research model depicted in Figure 1 was tested using SEM.The statistical analysis software packages used to perform the analysis were SPSS AMOS 22.0 and SPSS Statistics 23.0.

Measurement Model
The measurement model was assessed in terms of individual item loadings, reliability of measures and internal validity.Reliability was assessed using Cronbach's alpha.When Cronbach Alpha is equal to or greater than .60,measurement's internal validity is high (Malhotra, 1999).In Table 3, the values are .98for YB, .95 for TTK, .95 for THM, .93 for EKE, .93 for T, .96for ISB, .96for SE and .88 for SG.These values show that the dimensions are reliable and the features that need to be measured are correct.Before beginning to test the hypothesis with a structural equation model, exploratory factorial analysis was implemented to determine the measurement size used in the model.The criteria of KMO > .50shows whether the sample size is adequate for factorial analysis and the value for  2 is significant (Tabachnick & Fidell, 2001).This means that the sample size used in this study is adequate.The values in Table 4 show that the data is appropriate for exploratory factor analysis.
At the end of the implemented exploratory factor analysis, factors need to be greater than 1 and the factor values of constructs must greater than .5 in order to find the most convenient solution (Tabachnick & Fidell, 2001).For this reason, constructs with factor values < .50 were excluded from the analysis.The results are shown in Table 4.

Structural Model
Before evaluating the relationships between the structural equation model and the research model, it is necessary to determine that the model is statistically valid.Goodness of Fit Index (GFI), Adjusted Goodness of Fit Index (AGFI), Normed Fit Index (NFI), Relative Fit index (RFI), Incremental Fit index (IFI), Tucker-Lewis index (TLI), Comparative Fit Index (CFI) and Root Mean Square Error Approximation (RMSEA) values are shown in Table 5.
Match between the system and the real world SE1-The language used is natural, since the terms, phrases, and concepts are similar to those used in my day-to-day or study environment.SE2-I am not confused by the use of terms.SE3-I am not confused by the way symbols, icons, or images are used.SE4-There is no jargon used ('jargon' means words, acronyms or expressions that are developed and used by a group of people).SE5-The metaphors used correspond to real-world objects or concepts, for example, the icon for saving looks like a floppy disk.SE6-Information is arranged in a natural and logical order. .

Consistency and Standards
ISB4-Same words, phrases, situations, or actions refer to the same thing throughout the system.ISB5-Colours are used in a consistent manner (same way) throughout the system.ISB6-ISB6-Graphics, icons and images are consistently used throughout the system.ISB7-There is consistency in the screen layouts.ISB8-There is consistency in the use of menus ISB9-There is consistency in use of font types and sizes.ISB10-Links to pages are consistent with the titles of the pages they link to.
. 676 .696 .719 .753 .739 .726 .739Recognition rather than Recall T1-Instructions on how to use the system are visible.T2-There is an obvious relationship between controls and their actions.T3-Objects to use, such as graphics on tools bars, are visible. .

.845 .784
Flexibility and efficiency of use EKE1-The site caters for different levels of users, from novice to experts.EKE3-The site guides novice users sufficiently.EKE5-The system is flexible enough to enable users to adjust settings to suit them, ie customise the system.

Aesthetic and minimalist design
.703 .755 .741 THM2-The information on each page is not too much to confuse or distract me.
.579 THM3-There are no excessive use of graphics and images on the site.
.648 THM4-Dialog boxes provide adequate information for performing tasks.THM5-Dropdown lists and menus have the required options to choose from.

Recognition, diagnosis, and recovery from errors
TTK1-Error messages are expressed in plain language.TTK2-Error messages indicate precisely what the problem is.TTK3-Each message gives a procedure to fix the error.TTK4-The procedure to fix the error is specific, quick and efficient.TTK6-The site provides for easy reversal of action where possible, for example, by providing both Undo and Redo.

Help and Documentation
YB1-I find the help facilities -such as online help and the glossary -useful.YB2-The help facilities are easy to use.
. According to the results in Table 5, the value obtained by the χ2 value divided by degrees of freedom is equal to 2.29.When this value is between 0 and 3, the data and model are valid.When this value is equal or less than 5 it is still acceptable (Hooper, Coughlan, & Mullen, 2008).Since χ2 value is sensitive to sample size, other measures related to the structural equation model were also studied.According to this, the other measures (GFI = .76;AGFI = .72;NFI = .85;RFI = .84;IFI = .91;TLI = .90;CFI = .91)are all within acceptable values.When the values are close to 1.0, it shows that the data is acceptable for the model.According to those criteria, the data is compatible with the research model.Moreover, the RMSEA value, which must be between .05 and .10,was found to be .07.According to the RMSEA value, it can be said the data is compatible with the research model.These results show that the sample size is adequate for the research model, and the model is significant and valid statistically.
According to the Hoelter .05index, the minimum sample size is 113 to test research hypotheses at the .05significance level, whereas for the Hoelter .01index the minimum sample size is 117 to test research hypotheses at the .01significance level.Our sample size to test the research hypotheses is larger than the minimum required sample size.The obtained structural equation model by required modifications is shown in Figure 2, in which the arrows show the relationships between variables.The values on those arrows are the standardized regression weights.

Conclusions
The main task for the student is learning and usability should not be a barrier to student's learning since it can impact motivation.E-Learning usability is challenging since it has many components such as platform, course, content coming from different sources such as authors, instructors, developers.
In this respect, this study examined the critical factors of usability evaluation of the learning management system Moodle as part of students' blended learning in a computer literacy course at a funded University in Turkey.The model based on Nielsen's 10 heuristics and was tested with data from 236 students.Aspects like help and documentation, recognition, diagnosis, and recovery from errors, aesthetic and minimalist design, consistency and standards, match between the system and the real world, recognition rather than recall, flexibility and efficiency of use, visibility of the system status were taken into consideration in the research model according to the findings after an exploratory factor analysis.New model explains 71.82 percent of the variance which is indicative of strong explanatory power.
The results showed that users generally agreed with the system (60%).Some of the usability problems were identified were as follows: Heuristic 1 Visibility of system status 1. Lack of feedback.2. Pace of feedback is slow.
3. Difficulty in finding the object.Heuristic 2 Match between the system and the real world 1. Confusing.
Heuristic 7 Flexibility and efficiency of use 1.The system does not guide novice users.Heuristic 8 Authenticity and minimalism in design 1.Too many information.Heuristic 10 Help and documentation 1.It is not easy to search for information on the site.2. System is slow.

Dimensions
In this study, first exploratory factor analysis was used to develop the measurement model; then, the structural model for the research was tested using SEM.From the SEM analysis, eight variables are proven to have critical relationships with Usability Evaluation.Help and documentation, recognition, diagnosis, and recovery from errors, aesthetic and minimalist design, consistency and standards, match between the system and the real world, recognition rather than recall, flexibility and efficiency of use, visibility of the system status were found significant and the model explains 71.82 percent of the variance which is indicative of strong explanatory power.
Learning management systems, like any other systems, need to be monitored and maintained constantly to prevent usability problems that students can face.The quality and the structure of the system must facilitate student use of LMS.Planning needs to be given the importance and strategies must be applied.Uploading real life course content, offering practical information, online announcements, discussions and providing online exams and feedback as lecturers, will focus students' attention to website and the ease of use will motivate them.Moreover, LMS should not be seen only as a technological tool but also learning practices are important part of it.Therefore, if the usability evaluation captures the pedagogical effectiveness then it will succeed and be widely accepted.
Findings of this research will be valuable for both academics and practitioners of eLearning systems.Since the population of the study is limited to freshman students who enrolled in the computer literacy course, the model should also be tested with respect to other courses at the university.Therefore, further research is necessary.

Limitations of the Study
In this study, the main limitation was that the population was limited to freshman students who enrolled in the computer literacy course at the university and the sample was limited to students who chose to complete the survey.The study assumed that the respondents answered all survey questions honestly.

Figure
Figure 1.Research model

Figure 2 .
Figure 2. The standardized coefficients and the R2 values

Table 1
Demographics of Survey Participants

Table 2
Background of the Survey Participants

Table 3
Variable Definitions and Measurements

Table 4
Exploratory Factor Analysis Results

Table 5
Measurement Model's Goodness of Fit values