The best aspect of the ChatGPT responses to AsiaGlobal Online’s queries is that they were reasonably well organized and written with good grammar and spelling. Only a couple of times did our editor have to correct an awkward phrase. The artificial intelligence appears to answer questions in a very human way by first repeating the question and then providing its response in a structure that closely hews to the way the request was made.
What is missing are references and sources – and indeed, it appears that the chatbot is not above providing information that may not actually be accurate or true. In a few instances, the replies referred to the signing of a joint declaration in a certain year or the holding of a particular workshop, even providing its title, but when the editor tired to verify the existence of the document or a record of the event, this was not possible. In one sentence, the bot referred to members of NATO in the Indo-Pacific, making it unclear if it meant a NATO country that had a presence in the region or might actually be located in the region (there are no NATO members in the Indo-Pacific, of course). The editor removed any information that was questionable or unverifiable.
The analysis itself was not especially deep or complex. An international relations professor might give the paper a low passing mark, especially as it lacked any references and sourcing. If reckoned to be an academic work or an opinion piece for a newspaper, a teacher or editor applying moderate standards would find the essay superficial, possibly the product of a cursory internet search or two without any significant research. That it included incorrect information would ensure a failing mark or rejection if properly fact checked by the assessor. As an interim policy, The University of Hong Kong prohibits the use of ChatGPT or any other AI-based tool for all classroom, coursework and assessment tasks.
The editor asked ChatGPT to provide three suggestions for further reading, which AsiaGlobal Online normally appends to the end of every published piece. The bot proposed three papers, any references to which the editor could not find online, suggesting that these articles may not actually exist.
Separately, the editor conducted another exercise – asking ChatGPT to suggest outstanding possible candidates to take over a major non-governmental institution and think tank on Asian affairs that is currently looking for a new leader. The bot proposed three names – one a plausible figure, while the other two might have been good picks if they were alive! The artificial intelligence is not up to date on developments since 2021.
To close out the test of ChatGPT’s analytical ability, AsiaGlobal Online asked it to comment on the NATO secretary general’s visit to the Indo-Pacific from Beijing’s perspective. This was its response: