Evaluating an Information Resource
Let's walk through the steps of the ROBOT test using ChatGPT 3.5 as an example.
Reliability: Where does the content in the tool come from? How reliable is that content?
According to OpenAI's FAQs about ChatGPT, this content comes from "three primary sources of information: (1) information that is publicly available on the internet, (2) information that we license from third parties, and (3) information that our users or human trainers provide" ("How ChatGPT and Our Language Models Are Developed", n.d.). The content that ChatGPT provides is only as reliable as the information that goes into it, and should be double-checked against other sources.
Objective: What is the tool's stated purpose? What is it meant to help you accomplish?
ChatGPT 3.5 is meant to be a text generator. It can "organize or summarize text, or...write new text" ("How ChatGPT and Our Language Models Are Developed", n.d.).
Bias: What types of bias might be present in the information the tool provides? Are there ethical issues associated with this technology? Does the tool acknowledge bias and/or ethical issues?
OpenAI acknowledges that "the model is skewed towards Western views and performs best in English," and that "the model's dialogue nature can reinforce a user's biases over the course of interaction" ("Is ChatGPT biased?", n.d.). Given the types of algorithmic bias we already know exist on the internet, it's likely that ChatGPT would reflect and reinforce that bias since it gathers content directly from the web.
Owner: Who developed this technology? Who owns it? Who has access to it? Does everyone have equal access? What happens to information you enter into the tool?
OpenAI, the company that developed and owns ChatGPT, says that its mission "is to ensure that artificial general intelligence benefits all of humanity" ("About", n.d.). ChatGPT-3.5 is freely available to anyone who creates an account, but the more advanced GPT-4 is only available via paid subscription. In order to create an account, a user needs to provide personal information, including a phone number. Data and content that users provide during conversations with the chatbot may be used to further "train" the model.
Type: How does it work? Is it a large language model or another type of technology?
ChatGPT is a large language model that is trained on an incredible amount of content, meaning that it has "learn[ed] how words tend to appear in context with other words. It then uses what it has learned to predict the next most likely word that might appear in response to a user request, and each subsequent word after that" ("How ChatGPT and Our Language Models Are Developed", n.d.).
About. (n.d.). OpenAI. Retrieved February 16, 2024, from openai.com/about
Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test
How ChatGPT and our language models are developed. (n.d.). OpenAI. Retrieved February 16, 2024, from help.openai.com/en/articles/7842364-how-chatgpt-and-our-language-models-are-developed
Is ChatGPT biased? (n.d.). OpenAI. Retrieved February 16, 2024, from help.openai.com/en/articles/8313359-is-chatgpt-biased