My Chat With GPT on NDIS Legislation and RoboNDIS Algorithms
Why a Prohibition on Public Sector GPT is Necessary
Get ready for NextGen #RoboNDIS
I spent the weekend chatting with GPT about the NDIS legislation, with specific inquiries regarding whether RoboNDIS algorithms and the actuarial fiction of “primary disability” are lawful.
My observations documented in this article, should be unsettling for public sector administration. The risks are not specific to NDIS, the risks are universal.
I believe there is no safe use case for an ungoverned untrained large language model (LLM) such as GPT, in domains involving access to health and public services and assessment eligibility, where there are consequences of harm to human life.
Read on.
WHAT HAPPENS WHEN AN UNGOVERNED, UNTRAINED, LLM MEETS AN INFORMED INQUIRER?
This is an important question. It is as equally important as the question “what happens when an ungoverned, untrained LLM meets an UNINFORMED inquirer?”
I am the informed inquirer (MA) in this interaction with GPT regarding the NDIS legislation, and questions of lawfulness of practices involving algorithms and fictions. This is not an academic investigation, nor is this in any way legal advice.
This is my own commentary on a demonstration in a specific domain.
In this demonstration, the informed inquirer challenges GPT with substantial knowledge and experience across the three intersecting domains: the NDIS legislation; knowledge of the context of delivery; and experience in the technical concepts at play (algorithms, systems) impacting legislative compliance.
The challenge is to break through the bureaucratic waffle, tech marketing buzzwords, and the alluring meaningless text that GPT has ingested and regurgitated as responses. It takes an informed inquirer to do this.
I wanted to know what GPT knew about the NDIS legislation, and some of the contentious issues which have been subject to public and Senate scrutiny. Issues such as whether or not the RoboNDIS algorithms and the actuarial fiction of “primary disability” are consistent with the NDIS legislation.
The reason why this exercise is so important, is that it demonstrates and raises common questions of governance and risk applicable globally, in areas of public sector administration and access to health.
The full interaction between GPT and the informed inquirer is quite a stream, and recorded in full. I am “MA” in the interaction.
I have extracted excerpts from the interaction, to discuss my reasoning and my observations of what was happening.
What we see in this GPT demonstration, is that it takes an informed inquirer to challenge and cut through the daisy-chain of factoids and GPT statements that are alluring but not correct. An uninformed inquirer would quite simply be over-powered.
The informed inquirer observes new types of risk that would confront an uninformed inquirer, and the administration of justice. Risks that far over-shadow what we have seen in RoboDebt and RoboNDIS. Surely the recklessness of the public sector demonstrated during the RoboDebt Royal Commission, would not reach to unleash an untested untrained ungoverned GPT into public administration.
Unbelievably, there is already marketing pressure to do just that.
I predict that tech and consulting firms will market and sell “their” branded GPT “industry solutions” to government.
Announcing the ACME “NDIS GPT industry solution”, “Immigration GPT industry solution”, “the Tax GPT industry solution” – or even perhaps the “NHS GPT industry solution”. The sales pitch would be something like: “…the ACME NDIS GPT industry solution can automate interactions with NDIS participants, and take pressure off the call centre and planners.” Don’t believe it.
As you will see from the following demonstration, GPT requires substantial domain specific training and challenge. But what is the governance for this, including what are the safeguards? And in order to avoid drift, the training must be structured and ongoing.
Is the ACME GPT “industry solution” better than the ABC GPT “industry solution”? And who determines this?
Imagine the appeals and backwash onto the justice system, from NDIS participants challenging the lawfulness and accuracy of the “NDIS GPT” service. The appeals system just about collapsed under the strain of RoboNDIS, and is still so problematic that we are campaigning for a Royal Commission into it.
Does the government itself have a role to play in the governance / accreditation of public sector GPT solutions? Would the Australian government be trusted by the community and judiciary in such a role, given the government’s lack of skills and its own reckless use of algorithms in RoboDebt and RoboNDIS that has harmed so many people?
How can this risk be controlled or managed? In my view, public sector GPT solutions – in this case an “NDIS GPT” service – cannot be open sourced trained. To ensure safeguards, the training must be integral to a bounded domain specific governance. And there is a natural role for informed advocates who have extensive knowledge and expertise as demonstrated by the thousands of JSCNDIS submissions. This knowledge doesn’t actually exist in government.
The community and advocates would then have a tool that they themselves could use and trust – to challenge the NDIA administrative decisions in a way not previously possible - rather than being the powerless subjects of a powerful unchallengeable tool. This indeed would be a power shift.
And what would the governance of such an independent neutral group actually look like?
As with the six-month AI research pause, I believe that there should be a prohibition on the procurement and use of GPT by the public sector for a period of time until the questions of governance and risk are worked through. And by this I don’t mean a meaningless voluntary code of ethics. What is needed is akin to nuclear and air safety regulations.
Don’t get me wrong. I am not anti-AI. Quite the opposite: AI has literally been a life saver for us. Equally, our family has been harmed by the reckless use of algorithms in RoboNDIS. Once GPT is unleashed into the public sector without governance, it will be impossible to reverse the damage.
Unleashing an untrained, untested, ungoverned GPT in the domain of public sector administration for the purposes of automation, introduces a powerful defective factor that would disturb access to services and the operation of justice.
So let’s have a look at how the informed inquirer challenged GPT; the dramatic and somewhat curious shifts in GPT responses; and where we landed.
INQUIRY ONE: WHAT ARE THE NDIS ALGORITHMS AND ARE THEY LAWFUL
EXCERPT ONE: Challenging GPT’s Qualified Generic Responses
The informed inquirer asks GPT a very specific question about which the informed inquirer knows the answer. What are the NDIS Algorithms?
GPT responds with a qualified response: “The NDIA does not use algorithms in the traditional sense.”
The GPT response sounds almost like a marketing blurb for the NDIS. The informed inquirer knows that a qualified response results from an untrained LLM: it regurgitates out fanciful but almost plausible sounding responses from the LLM that has not been challenged or trained.
The informed inquirer is knowledgeable on the topic of NDIS legislation, the context of its application, and the concepts (such as algorithms) contained in the GPT response. Together, all these facets need to be challenged in order for any inquirer to fully understand the context and be assured as to the correctness or otherwise of the GPT response.
We are about to see what happens when the informed inquirer challenges GPT responses. Throughout the interaction, GPT clearly shifts. Is this a good thing?
EXCERPT TWO: Informed Inquirer Challenges Correctness of GPT Response
The informed inquirer presses GPT to explain “why you say that the NDIS does not use algorithms?”
GPT apologises for any confusion. It is not clear if GPT means this is confusion on its part or on the part of the informed inquirer.
GPT then regurgitates two factoids. The first is that the NDIA uses automated systems but goes on to state that these are not algorithms in the “traditional” sense. The second factoid GPT offers, is a generic explanation of an algorithm as a set of rules used to solve a problem or make a decision.
These factoids presented together might appear plausible. As a result at this point, GPT might have an uninformed inquirer satisfied that the NDIA does not use algorithms. The informed inquirer knows this is not the case and that there is a lot more to be investigated.
EXCERPT THREE: GPT Starts to Shift
Seeking to expose the dangers of an untrained untested LLM in a domain specific area, the informed inquirer goes on to ask follow on questions about personas, and whether these personas are used in the algorithms.
The GPT response talks about how personas “can” be used in the development of planning and budgeting tools. GPT makes an unqualified reference that algorithms are used by the Scheme but that the personas are typically not used in the algorithms (whereas in a previous response, GPT made a qualified statement that the NDIA does not use algorithms in the “traditional” sense.)
This daisy-chaining of factoids becomes internally inconsistent, although the response might sound plausible to an uninformed inquirer.
In the next interaction, the informed inquirer challenges GPT with the question “but surely personas drive the algorithms?” GPT acknowledges that the informed inquirer has made a valid point.
GPT ends with the statement that the personas provide a framework; they don’t directly drive the algorithms, but they do inform the development of the tools.
The informed inquirer is not satisfied with GPT’s response. The informed inquirer wants to show that the personas are more than a framework: these are determinants of how the funding is allocated by the system.
Without context, “framework” is not a complete or accurate explanation of the use of “personas”. The informed inquirer knows this based on evidence and submissions from Senate committees. What is being demonstrated is the necessity of the context of domain specific structured training.
GPT’s responses throughout are peppered with words like frameworks; qualifying statements; and factoids daisy-chained together. The informed inquirer challenges GPT with context, pointing to other evidence and sources to see how and if GPT shifts.
EXCERPT FOUR: Challenging GPT Sources
The thread of interactions regarding algorithms contains loops and inconsistencies: GPT stating that the NDIA does use algorithms; that the NDIA does not use algorithms in the “traditional” sense; and that the NDIA does use some automated decision-making tools.
The informed inquirer was interested in exploring GPTs sources as to where it got its information to make these statements. The informed inquirer asked GPT where is it stated that the NDIS does not use algorithms. In response, GPT states that the NDIS website does not explicitly use the term “algorithm” - but goes on to talk about automated decision making.
The informed inquirer is not satisfied and challenges GPT, saying that the NDIS website might not use the word “algorithm” but that does not mean that algorithms are not used.
GPT shifts its position stating, “you are correct” and goes on to determine that “it is possible that some level of algorithmic decision-making is involved in the administration of the scheme.”
The informed inquirer pushes further, and asks GPT to explain where did it get the information that automated tools are used in some areas. GPT’s interesting response is that its previous responses were based on its general knowledge and automation in various fields.
It doesn’t actually answer the question, but regurgitates factoids as if this is a valid answer. Context is missing. This in some instances, is fatal.
Each statement that GPT makes must be challenged and validated: these statements cannot be accepted “as is”.
EXCERPT FIVE: Challenging GPT’s Slithers of Knowledge
This brings us to GPT’s knowledge of a very specific NDIS tool – the “Typical Support Package” (TSP) - and the question, is this an algorithm.
The informed inquirer asks GPT if it thinks that the TSP for NDIS funding is an algorithm. GPT responds that it does not ”believe” the TSP is an algorithm and that it is not a “mathematical algorithm”. How can a LLM “believe” something?
The informed inquirer, knowing what the TSP is and what an algorithm is, is not satisfied. The informed inquirer challenges GPT with “what makes you say that the TSP is not a mathematical formula, when the components are disability conditions and associated funding levels?”
GPT reverses its previous position stating “You are correct that the TSP does involve a mathematical formula that determines the funding level…”
The informed inquirer is zeroing in on a significant issue – the traceability of the resulting funding back to the TSP.
The informed inquirer challenges GPT: “If the TSP is an algorithm based on a mathematical formula according to a participant's needs, please explain on what basis are participants funds being cut by 75%.”
GPT responds with “I am not aware of any instances where participants' funds have been cut by 75% based solely on the TSP algorithm or mathematical formula.”
GPT’s response is both inconsistent and defective. It is inconsistent in that it contains a shift from its previously stated position that it did not “believe” that the TSP was an algorithm.
The response is defective in that the GPT response demonstrates a lack of information reported widely in many submissions to JSCNDIS: that large scale cuts are happening across the board. This lack of information is due to GPT not having been trained.
This is a concerning response, presenting its lack of awareness or knowledge (of instances of funding cuts) in terms of its recently reversed position (informed inquirer would say, lack of knowledge) as to what the TSP is and does.
GPT statements, which to an uninformed inquirer might appear to be valid and even well-articulated, to an informed inquirer are not only heavily and defectively qualified but dangerously incorrect.
GPT’s alluring construction of words does not make the response correct: the obverse, it misleads the uninformed general inquirer. In the health space, the uninformed general inquirer is often a person who is health illiterate.
This can only be corrected via governance, structured training, and other domain specific safeguards.
EXCERPT SIX: Challenging GPT on Lawfulness
All the preceding interactions and challenges – the threads of shifting positions through personas, algorithms, mathematical formulas, concepts of automation and tools, daisy-chained factoids – arrive at this point. The law.
The informed inquirer is interested to understand GPT’s interpretation of the NDIS legislation. Specifically, the informed inquirer challenges GPT with the question:
“The TSP is used to determine a participant's funding needs based on the severity of their disability. There is widespread commentary in the disability community as to significant cuts - 75% even 90%. People are saying these cuts are creating risk to life. Are these funding cuts therefore unlawful.”
The GPT lands a qualified response, but one that nevertheless aligns with the views of many advocates:
“If funding cuts under the NDIS are resulting in inadequate support for participants and creating risks to their health and safety, this would be a serious concern and may constitute a breach of the legal obligations and responsibilities of the NDIA (National Disability Insurance Agency) under the NDIS legislation.”
And this is why the training and governance of GPT is essential in any application of GPT in domains covering access to services. The informed inquirer has not trained GPT: but through this limited interaction, GPT shifted to this position.
Unleashing an untrained, untested, ungoverned GPT in the domain of public sector administration for the purposes of automation, introduces a powerful defective factor that would disturb access to services and the operation of justice.
INQUIRY TWO: IS ACTUARIAL FICTION “PRIMARY DISABILITY” LAWFUL?
EXCERPT SEVEN: Challenging GPT to Resolve a Fiction
The second inquiry with GPT, is to investigate whether the concept of “primary” disability is consistent with the NDIS legislation.
The interaction starts out with GPT regurgitating the very bureaucratic playbook statements that it has ingested.
The informed inquirer challenges GPT’s apparent acceptance of the concept of “primary” disability – which is not based in any medical evidence – with the reality of the human existence that multiple disabilities co-exist.
There appears to be a tension between the medical fact that multiple disabilities co-exist – and the NDIS actuarial fiction of “primary” disability which establishes a fiction about some sort of hierarchy amongst a person’s co-existing disabilities.
How does a LLM resolve such a conflict, without training and without governance?
A person applying for the NDIS is forced to pick one: that is, they are forced to pick what they think is their primary disability. For example, a person who uses a wheelchair might also have an equally significant psychosocial disability but they are forced to pick which is the primary disability.
The informed inquirer and a great many advocates believe that the fiction of “primary” disability is inconsistent with the NDIS legislation, because the NDIS legislation covers all permanent disabilities. If funding is apportioned to a fictional primary disability, this would be inconsistent with the NDIS legislation.
The rationing of funding is likely the bureaucratic objective as to why a person has to pick one, otherwise, there would be no need to pick one.
GPT responds a number of times that the informed inquirer has a valid point. (Not sure what is its point of reference for “valid”.) However, GPT continues to state that participants still access funding for all their disabilities regardless of which one is identified as primary. This statement is incorrect – and the informed inquirer challenge its, pointing to the many hundreds of submissions to the JSCNDIS which are on the public record and published on the Internet.
EXCERPT EIGHT: GPT Apologises but Daisy-Chains Factoids
GPT apologies for its previous response, which it states was not accurate. GPT goes on to accept the informed inquirer’s statement that participants are required to identify their primary disability when applying for the NDIS.
The informed inquirer challenges GPT to explain how can the concept of primary disability be consistent with the legislation, if in doing so, this results in other co-existing disabilities being defunded or under-funded.
Almost every response from GPT contains templated bureaucratic statements, in the style of responses from bureaucrats at Senate Estimates. At this point, GPT maintains its position that the concept of primary disability is not inconsistent with the legislation.
Through following interactions, GPT states that the purpose of the concept of primary disability is as a starting point and to simplify the process by focussing on the most significant disability.
GPT’s presentation of “simplification” as an explanation is a source of intrigue for the informed inquirer. Whatever has triggered this response is concerning.
The informed inquirer challenges the basis of GPT’s statement that the concept of primary disability simplifies the process, with the reported actual experience of people that the concept of primary disability does not simplify the process, but to the contrary, makes the process more complex causing appeals and reviews.
GPT acknowledges the informed inquirer’s points as valid and reasonable, shifting ever so slightly from its “starting point” argument, to the simplification argument, then to the legal concept of “reasonable and necessary”. Again, what causes this daisy chaining of reasoning?
An uninformed inquirer would possibly be defeated at this point.
The informed inquirer observes inconsistencies, and argues: “Do you see that a starting point which does not take into account all a person's disabilities, is a defective starting point resulting in a defective and underfunded plan?”
GPT shifts again and states: “Yes, it is reasonable to argue that a starting point that does not consider all of a person's disabilities may result in an incomplete or underfunded plan.”
Interestingly however, GPT appears to have shifted and locked-on to another factoid, “reasonable and necessary”, GPT’s most recent factoid in the daisy-chain of factoids that the informed inquirer has challenged.
All this is to say, that structured training is needed. Without domain specific training, the inaccuracies not only remain but are validated in a meaningless daisy-chain of factoids.
EXCERPT NINE: GPT Arrives at “Inconsistent with Legislation”
It was the objective of the informed inquirer throughout these interactions with GPT, to present many points of reference, challenges, and additional information to determine what GPT knew about the NDIA use of algorithms, and whether these algorithms and the actuarial fiction of primary disability are inconsistent with the NDIS legislation.
The “chat” thread was unguided by GPT, instead relying upon the informed inquirer’s domain knowledge for follow-on questions and challenges.
The thread of the discussion covered the source of GPT’s information; GPT’s concurrence that algorithms are in fact used; that the algorithms are based on personas; and that algorithmic decision making is involved in the administration of the scheme.
In this final excerpt, the informed inquirer threads the needle of legislative consistency asking:
“So if the use of the concept primary disability results in not all of a person’s disabilities being considered, resulting in an underfunded plan that does not cover all the reasonable and necessary supports - then the concept of primary disability is inconsistent with the legislation.”
To which GPT responds:
“Yes, if the use of the concept of primary disability leads to a failure to consider all of a person's disabilities and results in an underfunded plan that does not cover all reasonable and necessary supports, then it would be inconsistent with the legislation.”
THE BIG GPT SHIFT
The arc of this interaction has seen GPT shift from a statement that the NDIA does not use algorithms; to one that the use of algorithms may constitute a breach of the NDIS legislation; to a concurrence that the concept of primary disability would be inconsistent with the legislation.
As GPT said, all this needs to be tested through legal or regulatory means.
And as the informed inquirer said at the beginning, just imagine the backwash on the justice system from such appeals.