AI RELIANCE IN GRADUATE EDUCATION SURVEY
Timeline
3 Weeks
3 Weeks
Discipline
Statistics, Surveying, Insight Discovery
Statistics, Surveying, Insight Discovery
For
Quantitative Research
Quantitative Research
Tools
Qualtrics
R
Qualtrics
R
Problem Statement
Graduate students' increasing reliance on AI tools like ChatGPT may compromise their academic integrity, particularly by encouraging the use of shortcuts and reducing the frequency of fact-checking.
Justification
With the widespread adoption of AI tools like ChatGPT, students in higher education are finding more efficient ways to complete tasks such as research and writing. However, this efficiency may come at the cost of academic rigor. Research shows that students who frequently use these tools often prioritize speed over accuracy, neglecting crucial fact-checking steps (Aljanabi & ChatGPT, 2024). This behavior can diminish critical thinking skills, as students may rely heavily on AI-generated information without thorough verification (Krullaars et al., 2024). Additionally, AI systems sometimes produce inaccurate or misleading information, further exacerbating the risks of over-reliance (Schei et al., 2024). Addressing these issues is crucial to ensuring that AI enhances the academic experience without undermining integrity.
Research Questions
- Do students fact check chatGPT?
- Are students with more obligations (due to employment/school) less likely to fact check chatGPT?
Survey Results
"What is your primary reason for using AI tools like ChatGPT in your academic work?"
Responses:
1. To assist with research and gathering information: 9 (24.3%)
2. To help with writing: 4 (10.8%)
3. To clarify difficult concepts or explanations: 8 (21.6%)
4. To save time on tasks: 12 (32.4%)
6. I do not use AI tools for academic work: 3 (8.1%)
7. Other: 1 (2.7%)
Analysis
The data reveals that students primarily use AI tools to save time on tasks (32.4%), indicating a focus on efficiency. There's a fairly even distribution among using AI for research (24.3%) and clarifying concepts (21.6%), with less emphasis on writing help (10.8%). Notably, only 8.1% of students don't use AI tools at all, suggesting widespread adoption in academic settings. The overall distribution suggests AI is seen mainly as a practical tool for task completion, information gathering, and understanding complex concepts. The low percentage of non-users implies that AI tools have become an integral part of many students' academic workflows.
Rationale for Survey Design
In this case, we've chosen to use the Agree/Disagree response format (ranging from "Strongly Disagree" to "Strongly Agree") for the Likert-style questions because this format is highly effective in capturing subjective attitudes, beliefs, and behaviors. Here's the rationale for this choice:
Measuring Opinions Easily: The agree/disagree format is perfect for capturing students' feelings and attitudes about their behavior. It’s familiar, straightforward, and makes it easy for respondents to give an honest answer.
Consistency Across Questions: Since we’re measuring one big idea (academic diligence and fact-checking), using the same format for each question keeps things simple and ensures the data is consistent and reliable.
User-Friendly: Respondents don’t need to overthink their answers—this format is widely recognized, which helps with clarity and engagement, leading to better results.
Captures Subjective Experience: These questions ask people to reflect on their own habits. Agree/disagree scales are ideal for self-assessments because they let respondents say how much they agree with a statement about themselves.
Balanced Responses: Including a neutral option means participants aren’t forced to pick a side if they don’t have a strong opinion, leading to more authentic, accurate responses.
Measuring Opinions Easily: The agree/disagree format is perfect for capturing students' feelings and attitudes about their behavior. It’s familiar, straightforward, and makes it easy for respondents to give an honest answer.
Consistency Across Questions: Since we’re measuring one big idea (academic diligence and fact-checking), using the same format for each question keeps things simple and ensures the data is consistent and reliable.
User-Friendly: Respondents don’t need to overthink their answers—this format is widely recognized, which helps with clarity and engagement, leading to better results.
Captures Subjective Experience: These questions ask people to reflect on their own habits. Agree/disagree scales are ideal for self-assessments because they let respondents say how much they agree with a statement about themselves.
Balanced Responses: Including a neutral option means participants aren’t forced to pick a side if they don’t have a strong opinion, leading to more authentic, accurate responses.
Improvements of the Survey Design
Bias Toward AI Users and Pretesting to Catch Issues
To improve the survey design, we could start with an initial questionnaire that doesn’t exclude participants if the constructs being measured don’t apply to them. Instead, it should be more inclusive and ensure that everyone can continue regardless of their AI usage. Another improvement would be to create questions that are applicable to all respondents, such as asking about their general thoughts and opinions on AI usage for schoolwork, rather than focusing on specific actions related to AI.
Survey Adaptability for Non-AI Users
The current survey structure did not fully consider the perspectives of students who do not use AI tools, and these respondents may have found some questions irrelevant. Incorporate open-ended questions for non-AI users to understand their reasons for not using AI tools. Questions such as "What factors influence your decision not to use AI tools for academic purposes?" could help gather valuable insights from non-users, thereby improving the survey's adaptability and inclusiveness.
Leading Questions and Social Desirability Bias
Questions could be rephrased in a way that removes judgmental phrasing. For example, the shortcut question could be revised to something more neutral, such as: "How often do you use methods that save time when completing school assignments?" This rephrasing makes the behavior sound less negative, which may lead to more honest responses. Additionally, adding questions that frame shortcuts in a positive light (e.g., efficient time management) can help balance the perception and reduce bias.
References
Aljanabi, W., & ChatGPT, O. (2024). Exploring the impact of ChatGPT: Conversational AI in education. Frontiers in Education. https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2024.1379796/full
Krullaars, M., Ferrajão, J., & Ahmad, S. (2024). The effects of over-reliance on AI dialogue systems on students' cognitive abilities: A systematic review. Smart Learning Environments. https://slejournal.springeropen.com/articles/10.1186/s40561-024-00316-7
Schei, O. M., Møgelvang, A., & Ludvigsen, K. (2024). Perceptions and use of AI chatbots among students in higher education: A scoping review of empirical studies. Education Sciences. https://www.mdpi.com/2227-7102/14/8/922
Krullaars, M., Ferrajão, J., & Ahmad, S. (2024). The effects of over-reliance on AI dialogue systems on students' cognitive abilities: A systematic review. Smart Learning Environments. https://slejournal.springeropen.com/articles/10.1186/s40561-024-00316-7
Schei, O. M., Møgelvang, A., & Ludvigsen, K. (2024). Perceptions and use of AI chatbots among students in higher education: A scoping review of empirical studies. Education Sciences. https://www.mdpi.com/2227-7102/14/8/922