This episode explores the impact of AI on health technology assessment (HTA), focusing on whether AI makes work faster or better, and how to ensure credibility, transparency, and responsible use. Featuring Professor Gianluca Baio and Professor Rachael Hunter from University College London (UCL) and Matthew Bending and Jack Ishak from Thermo Fisher Scientific, the discussion covers validation, accreditation, industry roles, and future challenges in AI-enabled HTA.
This first episode of Thermo Fisher Scientific's The Real-World Evidence is Clear podcast features Matthew Bending (host), Vice President, Global Head of Health Economics & Market Access, Thermo Fisher Scientific; Gianluca Baio, Professor of Statistics and Health Economics, UCL; Rachael Hunter, Professor of Health Economics, UCL; and Jack Ishak, Vice President, Statistical Methodology & Innovation, Thermo Fisher Scientific.
AI is moving quickly into health technology assessment, but this episode focuses on the bigger question: is trust in the output keeping pace with the speed of adoption?
The discussion makes clear that in HTA, faster is only useful if the evidence remains robust, credible, and ready for scrutiny. The panel argues that AI should not be judged only by whether it saves time, but by whether it helps produce work that decision-makers can trust. That matters in a field where plausible is not enough and methodological rigor is everything.
A recurring theme is that AI can shift effort rather than remove it. Professor Rachael Hunter describes AI-assisted work that looked reasonable on the surface but still required detailed checking and correction. Jack Ishak makes a similar point in literature reviews: AI may help with abstract screening, but it is less reliable when extracting detailed evidence. Time saved early can easily be lost later in validation and quality control.
The episode also explores the risk of AI creating an illusion of competence. Professor Gianluca Baio argues that AI is most useful when it supports people who already understand the underlying methods. Used well, it can accelerate expert work. Used badly, it can allow people to move quickly without recognising when the output is flawed.
Transparency is another major theme. As AI becomes more common in HTA workflows, regulators and reviewers will likely expect clearer documentation of how it was used, what human oversight was applied, and how outputs were validated. The panel also considers who will shape the rules, with consultancies likely to move fastest, academia more cautiously, and HTA agencies remaining the ultimate gatekeepers.
Overall, the episode presents AI in HTA as promising but still unresolved. Adoption is already happening. The real challenge now is building the standards, governance, and skills needed to ensure that faster workflows still produce evidence that is rigorous, transparent, and trusted.
Matthew Bending (00:10): Hello and welcome to the Real World Evidence is Clear podcast. I'm Matthew Bending, the VP and Global Head of Health Economics and Market Access and I'm very pleased to be hosting the first episode alongside my colleague, Jack Ishak, who is our VP for Statistical Methods and Innovation.
So just to tell you a little bit about this series, it's about creating space for thoughtful conversations on the issue shaping evidence, access and decision making across healthcare. And for our first episode, we're beginning with a topic that is attracting a great deal of interest and certainly raising some very important questions for across our field.
So we're going to today build on the discussion, the great discussion we had at UCL in January of this year where we took an academic HTA and industry perspective looking at the role of AI in health technology assessment. And for me as an attendee there, one of the themes that really stood out there was this important tension. AI may be making components of HTA faster, but is it actually making the work better? More rigorous, more credible, and more useful for decision making.
It felt like a really good conversation to continue in our first episode. we're joined by two fantastic guests from UCL. We have Professor Gianluca Baio, who is the head of department and expert in statistics and health economics at UCL with particular expertise in Bayesian cost effectiveness analysis, causal inference, hierarchical modeling, and a longstanding contribution to advancing methods in health economic evaluation and we also have Professor Rachael Hunter who's joining us from the Policy Lab at UCL who leads the Health Economics Analysis and Research Methods team who specialises in the economic evaluation of complex interventions and bringing an important perspective on the realities of applied evaluation and policy relevant evidence. So, Gianluca and Rachael, thank you both very much for joining us. It's wonderful to have you today here and it really kicks us off on that first question. when it comes to AI and HTA, are we really making the work better or simply faster? And I'd ask, you know, for maybe a case example or a recent example that you've seen when you speak to that.
Gianluca Baio (02:36): First of all thank you for the invitation. It was really really good to do the event at UCL and it's really really good to pick it up and follow up here in the podcast. I think you make a very very good point there in trying to separate out the issue of whether we could do things quicker or better and do we actually do things quicker because I think and this has been for me a long-standing, I don't know what the best word is, maybe pet peeve, but a lot of what we see in terms of the tools that people use, particularly in our field, is a proxy for how sophisticated the modelers might be in terms of using the relevant methodologies. And obviously it's not like a one-to-one thing, it's not like if you use a tool then you're good and if you use a different tool then you're bad. But there is some level of sort of correlation there. And I think that is pretty similar or very much related to what you were saying. It can be like the Goldilocks of situations where we have, that we're using tools that enhance the way that we work and make it quicker. And that would be fantastic. Or it may be that actually we're doing things, assuming that we're going towards a shortcut, but we're messing things up because we're not really understanding the full details of what it is that we're doing.
Or we might in fact, taking a good shortcut and making things quicker, but still at the expense of some level of extra complexity that we might want to take into account and to take ownership of. So I think it is the key question, it seems to me. It is what we need to be really, really aware. And from my academic perspective, I see this as a big issue in terms of how we deal with our students, what we teach them. Rachael and I have involvement in one of our MSE programs, which is specifically devoted to health technology assessment and health economics and decision sciences. And again, I think it's a big deal now to try and teach the students how to use these tools, how to make a good relevant use of these things.
Matthew Bending (04:39): So Gianluca, are you saying therefore it's about science first, speed second, or both together?
Gianluca Baio (04:46): I think ideally we'd go both together, but it's very dangerous if we leave science behind thinking that we're running a shortcut and we have something that is very, very quick because we lose the full control of what we're doing. And again, if you do know what you're doing, then speed is really good. If you don't, it might get your car outside of the road and you don't want that.
Matthew Bending (05:05): Rachael, you're nodding as well. Do you agree on all those points?
Rachael Hunter (05:10): Yeah definitely, yes I mean it is the do you know what you're doing factor and that's where AI it offers a tool to potentially get you to your destination faster but the extent to which you kind of off-road it and know where you're going is a big question and if you don't know what you're doing then sure you get to the destination but what happened in the middle nobody quite knows.
Also, if you know what you're doing, for some of them, there are certain steps that need to be taken that can take a while, for example, data cleaning, or getting all of your information in together. But the actual analysis itself, if you know what you're doing, isn't going to take too long. It's the, again, the dangerous thing of tools that are being used by people where they don't know exactly what they're doing.
And the transparency then, is what Gianluca has highlighted, transparency of what the tool is doing and the ability to actually disentangle that and make sure it's doing it the correct way. That's the danger of where AI comes in. Just to give you ask for a case study, a recent example where I've been working with some people on, not going to give too many details You'll find out why in a moment. But working with colleagues who might be in the medical profession and they're wanting to do a bit of dabbling in health economics and they want to make an analysis go faster so they very nicely send you an analysis that was created with the help of Claude and as you can imagine you look at it and you go well this is about masters level student stuff I can't say it's specifically wrong in any way, but there's just little tweaks of it that just don't sit right. But because they've gone off and done it with Claude, it's going to take you just as long to go through and double check what they've done as it would take for you to do it yourself. And I think that's the regular problem we get into with AI of the double checking takes just as long as if you did it yourself.
Matthew Bending (06:54): I don't know, that's a great case example on there. And I suppose it's that double checking and the credibility piece as well, because I think there was a nature news story last week where a researcher had created a fake disease, put that into the publications, and then researchers had started to cite that disease.
And that kind of checking you say, Rachael, it's just so important. You've got to go through that, but not always checking the logic, but actually checking the provenance of the original publications within there. So, mean, Jack, I know from an industry, you know, often we want to do things quickly and that kind of speed is the key piece, like from getting medicines to market.
Jack Ishak (07:30): Yeah, that is true, but I agree that speed on its own is not enough. I think the value of AI is in being able to do it faster, but preserving the quality and maybe improving it where possible. Just doing it faster on its own. Ultimately, these are things that have to be decision grade evidence that are going to go into reviews. So very quickly, the value will disappear. And I like the example of AI writing the code, but then it's becoming difficult to check or edit. Once you take it out of the AI system and it's in your hands, it's like inheriting someone else's programs and deciphering what they're doing, how they're doing it can be very time consuming, possibly as much. And the other example that it makes me think of is in LitReview.
That's been a long standing use case where AI has been leveraged to help screen abstracts, for example. But then when you look at extracting information from papers, this is where AI starts kind of not being as powerful. And we've had examples where the time saved in the mechanical part of the extraction is kind of lost on the back end when they go and review and check and have to correct what's been done.
Matthew Bending (09:37): No, so Jack, for HTA bodies, and this kind of provides an opportunity, but a challenge, right, in the use of AI. And I wonder on reflection whether we think that HTA bodies are likely to hold the key to use here, or will academia and consultancy move first and then we'll see the HTA bodies moving. Rachael what do you think given the know the discussion I think we had on that?
Rachael Hunter (10:10): Yeah, think government bodies will always be more cautious and they will need that extra push. I personally think it's more likely to come from consultancy than it is academia because I think there is more of a time pressure there. Academia, we do move more slowly and we do like to double and triple check things. There are aspects of academia that obviously wanna push forward more and there's obviously work going into AI in terms of research. But I think overall it's more likely that they'll be pushed from consultancy and in particular thinking about where are our quality checks? How are we doing our quality checks? Where's the transparency? Where's the replication? Those kind of issues where they're gonna be pushed first.
Matthew Bending (10:54): So the more validation stage on there from that HTA piece. Gianluca what did you think?
Gianluca Baio (11:02): So I think in all the time that I've known Rachael, I can't remember a time that we haven't agreed on things. So, you know, I'll be a bit boring here, but I think the point of validation is key. I wonder whether we have an extra step as well, which might be some kind of accreditation, because again, to link back to what I was saying earlier on, we need a process where we're sure that people know what they're doing.
And then at that point, only at that point could we be relaxed about them leveraging the use of AI. Because again, if I am convinced that you know what you're doing and that even if you didn't have AI, you would do the right things in terms of modeling, for example, of using the data, of preparing what kind of analysis you need to do for that specific problem. Then at that point, I can be fairly convinced that you're doing the right things by taking the shortcut and asking Claude maybe to check your code or to fix your code to finalize to make it faster, for example. So I don't want some patronizing here. And obviously this would be a very complicated process and it would need, I agree, academia might be slower, consultancy might be quicker, but I think it would need a concerted effort from all the components that are involved here. And I wonder whether we should be working towards something that is, okay, let's make sure that we have a body of people who we trust can work on this and then move to the next level.
Matthew Bending (12:32): Okay, Jack are you in agreement, what do you think?
Jack Ishak (12:36): Should we be asking for accreditation now because when we submit things to agency for instance, the agencies evaluate the output. They have to figure out from what's being presented, how it was done, whether it was done properly and so on. There's no questioning of who did the work.
What was their background? How many years of experience and so on. So with AI, should that change? I kind of feel like AI is a tool in that process. In the same way, like the software that we use, you could use it properly and you can use it improperly and that has an effect on the result. In my opinion, the onus is on added transparency. I think we have to think about what does agency need to know about the process to evaluate the final product completely is the current standard of what has to be reported enough. So if I think, for example, of LLMs and the way one might use LLMs in helping support some of the tasks, should the interaction be submitted as an appendix, you know, some cleaned up version of that through reverse engineering and so on?
So in my view, I think there's a practical point where AI is very quickly becoming so pervasive, so spread out in the way it's being used, it becomes very tricky to manage where it's applied, where it's not applied, who's applying it, how. So I think it goes, we have to anchor it to the standard of what we're evaluating and what we need to know to do a proper full assessment.
Matthew Bending (14:16): So Jack, you're saying that the HTA agency, need to be the first movers in this on there because without that validation.
Jack Ishak (14:25): I think, yeah, I mean, in my view, the door is open. I don't think it's a question of whether they're gonna open the door. I think the door is open. I think we're waiting to see the rules of the game now and what the standards are gonna be and how they're gonna change when AI is involved.
Matthew Bending (14:43): And on that point, how do you see industry and consulting supporting that now?
Jack Ishak (14:48): I think industry and consulting play an important role there because ultimately I think they have to kind of align and they shape that to some extent, right? The way the work is done and what's being delivered shapes the standards at the same time. So I see it as a for stakeholder situation. I think agency does ultimately hold the strongest influence, they're the gatekeepers at the end of the day, so that they will set the rules. And I think industry and consulting will help shape that to align with ⁓ the everyday practice.
Gianluca Baio (15:23): I think, if I can interject, you said something that is very interesting, Jack, where you indicated eventually AI is just a tool. And I really agree with that. The problem is that it is a very, very powerful tool. So it's kind of the revolution that Google made when you stop having to go to the library to find the papers, except on steroids. it's several orders of magnitude bigger, I think, because also of the hype and the popularity.
So maybe I think the danger is that I do agree with you on general principles. This is just something else that we need to learn how to use. It's just that it hits us in the face a lot stronger than other things, I guess.
Jack Ishak (16:04): And very suddenly as well. It seems like an overnight we went from not having it to all of a sudden trying to use it for everything.
Gianluca Baio (16:06): Yes.
Matthew Bending (16:14): And linked to that, I mean we've talked about how AI helps in the process, the evidence assessment, but what about technologies that are frontier AI enabled that agencies are going to have to assess? I Rachael, what do you, practically from a policy piece with the speed that we're seeing here, how do we deal with that?
Rachael Hunter (16:45): I think that again, it's the government bodies and the approvers again, who are getting more stuck with this in terms of they've got exactly the same issue again with some of the the digital AI technologies coming through. Again, they want transparency. They want to be able to see the minutiae of it before they approve it. And again, we have that issue with AI of exactly what is it doing.
So, I think there's an issue of how it's approved and how it goes through the approval process. The other thing I've seen happening, which healthcare likes to do is some things that kind of sit just below the regulatory framework, they start to implement them anyway, even though there's limited evidence base for them. I think one of the tricky things with AI, particularly as a health economist has been much the same as, and Gianluca talked about, you when we have Google, we've had a whole range of productivity initiatives when we went to digital records as opposed to paper-based records.
When we started using emails, there's always that claim of, we're going to be more productive now, we're going to achieve more now. And then NHS is starting to make that claim with AI-based tools as well. Oh, there's going to be added productivity, we're going to do this faster, we're going to be able to diagnose people better. But we're seeing a similar issue to what we're seeing, we've been discussing here of, well, it sounds good, but he evidence base so far has been limited in terms of the efficiency aspect of it. Again, if you've got a tool, for example, where we're doing some of it within diagnostics, that yes, it does a better diagnosis, but you have to make the sample cleaner, more distinct. And there's a whole range of work that goes into feed it into the AI for them to be able to do the work. You will all of a sudden lose that efficiency.
And actually we're getting a replication of what happened with email, what happened when we went to digital, where sometimes we're actually having reductions in the workforce, we're seeing people trying to put AI in really quickly with actually no evidence that it is increasing efficiency. And I think that's actually where the key challenge for health economists and researchers come in is at what point do we put the brakes on innovation and say, come on guys, we need to check this first. At what point do we let it flow through and evaluate it, which then makes it a little bit trickier. I think that's a big health economics question for us with with AI.
Jack Ishak (18:56): I think with...
Matthew Bending (18:56): Yeah, go on Jack.
Jack Ishak (18:58): I was going to say, think with frontier AI, there's another layer of challenge, right? The frontier AI case is where AI is inherently tied into the technology that's being evaluated. And with the pace at which AI is changing, I think the challenge agencies are facing, what I'm evaluating today doesn't exist in a few months. It's completely different. And the data I had on its performance, is quickly outdated. So that presents a whole other set of challenges because the assessment has a very limited window. You know, the rules of when it needs to be reassessed, etc. have to be thought through. think the challenge is a bit less with the existing, the broadly used models, Claude and Chat GPT, etc. versions of the models that are used can be explicitly stated, those generally remain available, so that reproducibility is preserved to some extent, maybe it'll lessen over time, but Frontier presents a wholly other layer of challenge.
Gianluca Baio (20:05): And I wonder whether this actually means that we need to think about restructuring slightly the way that we think about the process. Because now it's not always like that, but it's not front and center that we say, here's some evidence, here's the evaluation. Do we know enough to make the decision? Should we wait until we get more evidence?
We want to make that decision, whether it's yes or no, for all sorts of good reasons. But I wonder whether, and again, I agree with you, Jack, whether this is a big change point where we should acknowledge more well based on the little that we know and it might change very very quickly at the moment it looks like a yes but what do you do now do you fully commit on the little that you know that it might change very very quickly do you take it in a way that it is kind of conditional or you know approve for a little while until we know more and I don't know the answer here but I think we should consider more seriously these kind of questions because again this is a complete different ballpark in terms of the evidence that can be generated.
Jack Ishak (21:06): In my mind, I think that issue comes down to augmentation versus automation. I think with augmentation, we have a certain level of protection, right? We always say the human is ultimately responsible. Whatever they're doing, they're accountable for what comes at the end. I think the leap will be the full automation of where almost everything is done by AI and the role of the human is reduced significantly.
And I think that'll be a much higher barrier to cross because then the issues of accuracy, how does it perform in this particular use case, kind of validating it to establish credibility may be a lot harder than in this augmentation framework where there's all the protections of the human in the loop and et cetera that that makes it a lot more palatable.
Matthew Bending (21:59): Okay, so you're saying automation is full AI, it's not just parts of the control, it's full and then augmentation is partly in the loop. What role, I mean in terms of the more complex, say health economic models, do you see that, how do you see AI developing in those areas?
Jack Ishak (22:01): Yeah.
Matthew Bending (22:21): How much of the more complex modeling or creative solutions can AI play a role in?
Gianluca Baio (22:29): That's a good question and I think it's probably wider than that. I think that again, like we were saying, AI can be a very, very good tool so it can expand the range of modeling capacity that you might have. I'm not fully convinced that in itself and alone can do that. And I go back to the point I was trying to make earlier on. I think what we should do the the biggest sticking point would be to ensure that the people who then use AI actually have a good knowledge and a good sense of what the very basic, in fact, and the good amount of methodology that is required and needed is there. But it opens up also an issue, I think, in terms of equity, perhaps, because what if you have access to a very good LLM, for example, or the latest version of a very good LLM?
And then you can leverage that. And what if you don't have access to that? And it may be in terms of geographical areas from where the dossier come from, or in terms of even your own industry. What if you're a richer company and you can leverage the better tools because you have more money to invest in that? Does that mean that your models are necessarily better? Or does that mean that you follow the shortcut better? So I don't know, again, I think I am agnostic in many of these things because I think it's very, very hard to have a formed opinion as things change completely. I think these are the key questions that we need to ask ourselves.
Rachael Hunter (23:54): So can I come in on that one? Because I think definitely the inequalities aspect, but also where I worry about academia falling a bit behind is again the tools as well. I can see industry and consultancy being faster at developing these tools and academia being more cautious and caution doesn't work well in this kind of space. So academia falling behind in terms of the analysis.
But I think also the other thing is the tools that we're using and what role they play. So traditionally, particularly when you're talking about nice, most models have been in Excel. Gianluca is obviously pushing forward with how we can utilize R and increase the use of that, which obviously functions better with AI than Excel does, although you can obviously do it with Excel. But it would be interesting to see the extent to which we see more.
So you guys probably heard of, you know, vibe coding and other things where people start to do more stuff in Python or other coding languages, particularly utilizing some of the big data that we're seeing coming out and how we leverage that to create very complex system-based models. I think those will be the pathways forward and those will be where it starts to get more difficult where you get.
Again, if people don't understand the coding or the background information, how it is that you then validate very complex things based on very, very big data that take a long time to run and only a certain number of people even have the skill base to be able to disentangle it. And I think this was raised at the away day. In the regulatory bodies, do they have the skill base there or have they all been nabbed by industry who can pay more and can actually attract that level of skill.
Matthew Bending (25:41): And Jack, how do you see the path for complex modeling?
Jack Ishak (25:45): I think even with the current capabilities of many of the large language models, there are elements of tasks and model development that can be helped. are certain things that LLMs are very good at that leverage the proper way can be beneficial. I coding is one example. I agree with the concern that things could get very complex and, you know, make it difficult for the end user to properly be able to evaluate, especially if they're doing it in areas that they're not comfortable with.
The distinction I always kind of try to make is, am I using the LLM to save myself time for something I'm able to do and then I can properly validate? Or am I using it to fill a gap, something that I'm not really comfortable myself doing and it's helping me learn how to do that, then I need to be very rigorous in that second category. And I need to think about how am going to check? How am I still going to own that final result at the end? So I think that's where the line has to be played in deployment. But in terms of the capability, if you think the other areas is validation, we haven't touched on yet, but you know, the AI doesn't get tired and lose focus and miss certain things. So in that sense, as it's a lower hanging fruit, perhaps, because the AI is not generating or creating, but checking something either the human did, or maybe the human did with AI assistance, but validation, catching the errors that humans may miss and checking for consistency and what you said you were going to do and what you're actually implementing in your code.
So there are areas like that where in the context of complex models can be really helpful, either as a first line tool or to supplement what the team is doing.
Gianluca Baio (27:37): If I can again interject, I totally agree here and Rachael mentioned the R-HTA bit for me, which obviously I feel very passionately about. I remember when we did our first workshop and it was back in 2018, I think, there was no sense. We had no idea what AI was or whether it was coming. It was like the stuff that you see in the sci-fi movies back then. And it feels like a million years ago, but it was only less than 10 years ago.
But even then, I think, and I go back to something I was saying earlier on, the point is exactly what you were saying. If you're comfortable doing it and it becomes a shortcut, then it's a whole different ballpark than the case where you don't know what you're doing and you're relying just on that tool. And it's the same with using Excel versus using R. It's not like you use R and you're better per se. It might become, if it's a proxy of you know what the model should be and then you use a tool that is suitable for that. But if you don't know what the model should be or how to do it, then it doesn't really matter whether you use AI or R or Excel or whatever. You just don't know what you're doing and that's very scary.
Jack Ishak (28:45): I think the idea that a powerful tool in the wrong hands can be harmful, even if it's a great tool and useful, right? I think a lot of this comes down to how you're using it, who's using it, and do they know what they're doing? I don't think you can, and this is where the discomfort is, I think, with trust around AI is who made the decision. I think it's when we get close to that decision-making point of is this you know, human driven or to what extent was it AI driven is where the discomfort starts happening. And I think that goes back to who's using it, how they're using it and what they're getting out of it.
Rachael Hunter (29:24): So one of my concerns along these lines is one of the best things we think we can find AI for is particularly the lower level tasks, the tasks that are repetitive and that we might get more junior level staff member to do. But actually AI does it perfectly fine and we can take out the really repetitive easy tasks. The issue is going to be how do we train people up then if there's no junior task for them to do. How do we make sure that they have the right skills and ability if we've kind of taken that earlier stage out and pretty much the only people who are doing it now are those who are experienced.
I think that's the issue of how do we make sure that doesn't end up happening again, how we don't just totally replace people with AI but ensure that they're part of the process so that people continue to build their skills to then be able to utilise it even better. And even then for some of us who've been doing this for many years, how do we make sure our skills are good enough to utilise AI in the right way as well? How do we actually fill that skill gap?
Matthew Bending (30:26): You raise the really important the train and develop and not only in the capabilities, but also around leadership capability and using AI from all walks from academia, consulting and industry. Gianluca, what do you think?
Gianluca Baio (30:46): I agree, I agree. And again, to make an example that is perhaps closer to home, we see this in how we have to deal with AI for the assessment of our students. There's an argument to say, when I have to mark 500 papers in June, I get very, tired after three, because it's a boring task. And so maybe it would be fairer to actually have an AI run through the first filtering through things.
But again, there's a sense in which that might be better, but you wouldn't leave it alone. And so maybe, again, the point is to not assume that we've got there already, because we definitely haven't. And so we need to train ourselves as well to learn how to use the tools and to combine the tools with the human capabilities. Because otherwise, again, there's too many dangers, I think.
Matthew Bending (31:39): and Jack, your thoughts.
Jack Ishak (31:41): I think there's an important challenge facing us in terms of the training of the next generation of researchers and scientists because traditionally, you know, we learn by doing, manually implementing what we're learning, especially when I think of statistics and health economics where there's manual part. I think that's going to change very quickly.
I think the next generation of students that are gonna come through are gonna be native AI users. They're not gonna know a world without AI. I see that in my own son who's in post high school and it's a ever present part of what he's doing. So I think that puts the impetus on us to think about what are the skills they need? How do you train? How do you develop the judgment, interpretation skills, the scientific rigor, without the doing part being at the core of it? And so I think there are a few key skills that become more important alongside the technical knowledge, the critical thinking, being more judgmental in what you're reviewing, how do you review and take ownership of something someone did for you.
So those become essential parts of development of both students as they come through university, but then also staff that come into consulting. And in that area, I think there might be an advantage, right? If junior staff coming through are comfortable and knowledgeable in AI, it frees up the time for their mentors to focus on developing their judgment skills. The why, why are you doing that particular analysis? Why are you doing it that way?
So I think there's opportunity there to again, you save time. What do you use that time for? If you can use it to enhance value, then that's the sweet spot of AI.
Gianluca Baio (33:39): It's very interesting because again it makes it difficult. I can be all self-righteous and I'm happy to do that and I tell the students that I know statistics better than they do and they should come to me and learn it from me but I can't tell them that I know AI better than they do because that's not true. They do it better than me in many ways. So it's difficult for us to be in that position.
Matthew Bending (33:57): Yeah, but lots of opportunity on the training and development right here. Massive, massive opportunity from just thinking about the future. And if we think about just to round up, if you think about the future in the next two to three years, what would each of you say responsible progress in AI enabled HTA? What does it look like? Rachael, let's start with you.
Rachael Hunter (34:24): You had to come to me first. What does the future look like? But this is the difficulty in what we're talking about. We don't know what the future is going to look like. So for me, responsible is I think what's been highlighted of accreditation, transparency, how it is that we push forward our innovation, but at the same time do it in a way where we acknowledge the limitations of AI.
I had someone explain AI to me and how it's being used in aspects of industry, particularly for coders is kind of the emperor's new clothes. There are problems with AI and nobody wants to talk about them. They're just putting it in there and hoping that it's going to work. And so in the future, I'm hoping that we do have those discussions of this is what AI can do and here are the benefits where it sits this week. Granted next week it might be better, but how do we check it and use it in a responsible way to make sure that we do push forward and we do utilize big data better. We do utilize the capabilities we've got in a better way and the intelligence we have in a better way. So yeah, that's my hope for the future of how we put processes around that aspect of it.
Matthew Bending (35:30): So having those open discussions on the credibility and transparency there. Gianluca?
Gianluca Baio (35:37): Yeah, I agree with that. And I think, you know, we always say that it would be important. It is important that we have a coming together of the three main components, academia and consultancy or industry and the regulators. I feel like this is the time where we can't just say that we really need to do it. Because again, it's pointless if one of the three components come up with the good solutions or takes the advancement forward in a way that the others don't catch up because that way everything gets broken. So my hope for the future, for the very next future is that we do take this opportunity to make it happen.
We've been saying it for a long time. We need to have more coordination, and it does happen. You know, it's not like the process is completely divorced and everybody does their thing. It's not like that, but this is bigger and stronger and quicker. And so we really need to get ourselves together, I think. And that's my hope.
Matthew Bending (36:32): And finally, Jack, what would you see in three years' time?
Jack Ishak (36:38): I would hope sooner than three years, but I think what is critical is clear and effective guidelines around AI use. I think that's a lot more challenging than guidelines for a lot of other things that have been developed in the past. I think there's a risk of it either being so restrictive that use of AI becomes almost impossible. For example, if you have to validate every single type of use case, then the effort to do that would exceed the benefit or might be too loose and then start clouding what we're doing. I think aligning on and it touches a little bit on John Lucas point of I think the stakeholders kind of aligning and coming together to shape that. I think that'll play a big role in getting the full value of AI.
Matthew Bending (37:26): So lots more discussions like this, Jack, in terms of what we're having and potentially larger conferences and as well events that are more focused around this topic on there. So I'm going to thank everyone for joining us for this first episode on there. In particular, thanks to Rachael and Gianluca and Jack for such a thoughtful discussion.
I think what's come through very clearly today is that the future AI in HTA is not just about speed, it's about trust, the credibility, the transparency and really having that discussion as an industry, academia and with our HTA bodies to really strengthen the quality of decision making. So we hope that the conversation has given you plenty to reflect on and we look forward to continue the discussion in some of future episodes.
Thank you everyone for listening today.