As artificial intelligence (AI) continues to reshape industries – driving business innovation, altering the labour market, and enhancing productivity – organisations are rushing to implement AI technologies across workflows. However, while doing so, they should avoid overlooking the need for reliability. It’s crucial to avoid the temptation of adopting AI quickly without ensuring its output is rooted in trusted and accurate data.

For 16 years, Stack Overflow has empowered developers as the go-to platform to ask questions and share knowledge with fellow technologists. Today, we are harnessing that history to address the urgent need to develop ethical AI. 

In setting a new standard for trusted and accurate data to be foundational in how we collectively build and deliver AI solutions to users, we want to create a future where people can use AI ethically and successfully. With many generative AI systems susceptible to hallucinations and misinformation, ensuring socially responsible AI is more critical than ever.

The Role of Community and Data Quality

The foundation of responsible AI lies in the quality of the data used to train it. High-quality data is the starting point for any ethical AI initiative. Fortunately, Stack Exchange Communities has built an enormous archive of reliable information from our developer community. 

With over a decade and a half of community-driven knowledge, including more than 58 million questions and answers, our platform provides a wealth of trusted, human-validated data that AI developers can used to train large language models (LLMs).

However, it’s not only the amount of data available but how it is used. Socially responsible use of community data must be mutually beneficial, with AI partners giving back to the communities they rely on. Our partners who contribute to community development gain access to more content, while those who don’t risk losing the trust of their users going forward. 

A Partnership Built on Responsibility

Our AI partner policy is rooted in a commitment to transparency, trust, and proper attribution. Any AI product or model that utilises Stack Overflow’s public data must attribute its insights back to the original posts that contributed to the model’s output. By crediting the subject matter experts and community members who have taken an active role in curating this information, we deliver a higher level of accountability.

Our annual Developer Survey of over 65,000 developers found that 65% of respondents are concerned about missing or incorrect attribution from data sources. Maintaining a higher level of transparency is critical to building a foundation of trust. Additionally, the licensed use of human-curated data can help companies reduce legal risk. Responsible use of AI and attribution isn’t just a question of ethics but a matter of increased legal and compliance risk for organisations. 

Ensuring Accurate and Up-to-Date Content

It’s important that AI models draw from the most current and accurate information available to keep them relevant and safe to use. 

While 76% of our Developer Survey respondents reveal they are currently using or planning to use AI tools, only 43% trust the accuracy of their outputs. On Stack Overflow’s public platform, a human moderator reviews both AI-assisted and human-submitted questions before publication. This step of human review provides an additional and necessary layer of trust. 

This human-in-the-loop approach not only maintains the accuracy and relevance of the information but also ensures that patterns are identified and additional context is applied when necessary. Furthermore, encouraging AI systems to interact directly with our community enables continuous model refinement and revalidation of our data.

The Importance of the Two-Way Feedback Loop

Transparency and continuous improvement are central to responsible AI development. A robust two-way communication loop between users and AI is critical for advancing the technology. In fact, 66% of developers express concerns about trusting AI’s outputs, making this feedback loop essential for maintaining confidence in the output of AI systems. 

Feedback from users informs improvements to models, which in turn helps to improve quality and reliability.

That’s why it’s vital to acknowledge and credit the community platforms that power AI systems. Without maintaining these feedback loops, we lose the opportunity for growth and innovation in our knowledge communities. 

Strength in Community Collaboration

At the core of successful and ethical AI use is community collaboration. Our mission is to bring together developers’ ingenuity, AI’s capabilities, and the tech community’s collective knowledge to solve problems, save time, and foster innovation in building the technology and products of the future. 

We believe the synergy between human expertise and technology will drive the future of socially responsible AI. At Stack Overflow, we are proud to lead this effort, collaborating with our API partners to push the boundaries of AI while staying committed to socially responsible practices.

Personalisation isn’t just a nice-to-have in consumer interactions — it’s a necessity. People want companies to understand them, and proactively meet their needs. However, this understanding needs to come without encroaching on customers’ privacy. This is especially crucial given that nearly 82% of consumers say they are somewhat or very concerned about how the use of AI for marketing, customer service, and technical support could potentially compromise their online privacy.  It’s a tricky balance, but it’s one that companies have to get right in order to lead their industries.

With that, I encourage organisations to lean into three key pillars of personalisation: AI, privacy, and customer experience.

1. The power of AI in personalisation

To tap into AI’s power to transform the way businesses interact with their customers, companies need to get a handle on their data first. The bedrock of any successful AI strategy is data – both in terms of quality and quantity. AI models grow and improve from the data they’re fed. As a result, companies need to have good data governance practices in place. Inputting small quantities of data can lead to recommendations that are questionable at best, and damaging at worst. Yet, large amounts of low-quality data won’t allow companies to generate the insights they need to improve services.

Organisations must define clear policies and processes for handling and managing data. This ensures that the data being used to train an AI model is accurate and reliable, forming the foundation for trustworthy personalisation efforts.

Another key to improving data quality is the creation of a customer feedback loop through user behaviour data. The process involves leveraging behavioural insights to inform AI tools and leads to more accurate outputs and improved personalisation. As customer usage increases, more data is generated, restarting the loop and providing a significant competitive advantage.

2. The privacy imperative

When a consumer interacts with any company today, whether through an app or a website, they’re sharing a wealth of information as they sign up with their email, share personal details and preferences, and engage with digital products. Whilst this is all powerful information for providing a more personalised experience, it comes with expectations. Consumers not only expect bespoke experiences, they also want assurances that they can trust their data is safe.

That’s why it’s so critical for organisations to adopt a privacy-first mindset, aligning the business model with a privacy-first ethos, and treating customer data as a valuable asset rather than a commodity. One way to balance personalisation and data protection is by adopting a privacy-by-design approach. This considers privacy from the outset of a project, rather than as an afterthought. By building privacy into processes, companies can ensure that they collect and process personal data in a way that is transparent and secure.  

Just as importantly, companies need to be transparent about where and how personalisation is showing up in user experiences throughout the entire product journey. Providing users with the choice to opt in or out at every step allows them to make informed decisions that align with their needs. This can include offering granular opt-in/out controls, rather than binary all-or-nothing choices.   

Regular privacy audits are also crucial, even after establishing privacy protocols and tools. By integrating consistent compliance checks alongside a privacy-first mindset, companies stand a better chance of gaining and maintaining user trust.

3. Elevating customer experience

The purpose of personalisation is driving incredible customer experiences, making this the third pillar of the triad. Enhancing user experiences requires a nuanced approach that goes beyond mere data utilisation. It’s about creating meaningful, contextual interactions that resonate with individual consumers.

Today’s consumers want experiences that anticipate their needs and provide legitimate value. This level of personalisation requires a deep understanding of customer journeys, preferences, and pain points across all touchpoints.

To truly elevate the customer experience, organisations need to adopt a multifaceted approach that starts with shifting from a transactional mindset to a relationship-based one, ensuring that personalised experiences are not just accurate, but timely and situationally appropriate. Equally crucial is the incorporation of emotional intelligence to deeply understand customers’ needs and  enhance perceived value. Furthermore, proactive engagement through predictive analytics allows brands to anticipate customer needs and offer solutions before problems arise. By combining these elements – contextual relevance, emotional intelligence, and proactive engagement – organisations can turn transactions into meaningful, value-driven relationships.

Looking at the whole personalisation picture

Mastering AI, privacy, and customer experience isn’t just important – it’s essential for effective personalisation. And these pillars are interconnected; neglect one, and the others will inevitably suffer. A powerful AI strategy without robust privacy measures will quickly erode customer trust. Likewise, strict privacy controls without the ability to deliver meaningful, personalised experiences will leave customers unsatisfied.

But achieving this balance is just the starting point. Customer expectations shift rapidly, privacy laws evolve, and new technologies emerge constantly. Organisations must continually adapt, using the data customers share to shape their approach; it’s about taking a proactive stance to meeting customers’ needs, not a reactive one.

Across the world, businesses are clamouring to adopt the latest AI technologies, and they’re willing invest significantly. According to Gartner, generative AI has produced a significant increase in infrastructure spending from organisations across the last few months, which prompted it to add approximately $63 billion to its January 2024 IT spending forecast. 

Capable of reshaping business operations, facilitating supply-chain efficiency, and revolutionising the customer experience, it’s no wonder major enterprises are keen to channel their budgets towards AI. But the benefits of AI can extend beyond large enterprises and make a considerable difference to small businesses too if adopted responsibly. 

Game-changing innovation 

Most SMBs don’t have the same ability for taking spending risks as their larger counterparts, so they need to be confident that any investments they do make are worthwhile. It’s therefore understandable why some might assume it to be an elite tool reserved for the major players.

To understand how SMBs can make the most of their AI investments, it’s important to first look at what the technology can offer. 

Across industries, AI is promising to be a game changer, taking day-to-day operations to a new level of accuracy and efficiency. AI technology can enhance businesses of all sizes by:

Enhancing customer experience

Businesses can use AI tools to process and analyse vast amounts of data – from spending habits and frequent buys to the length of time spent looking at a specific product. They can then use these insights to provide a more tailored experience via personalised recommendations, unique suggestions and substitution offers when a product is out of stock. And, with AI chat functions, businesses can provide more timely responses to any questions or requests, without always needing an abundance of customer service staff on hand. 

Powering day-to-day procedures

One of the most common and inclusive uses of AI across organisations is for assisting and automating everyday tasks including data input, coding support and content generation. These tools, such as OpenAI’s ChatGPT and Microsoft Copilot applications, don’t require big investments to adopt. Smaller teams and businesses are already using them to save valuable employee time and resources and boost productivity. This also saves the need for these organisations to outsource these capabilities where they might not have them otherwise. 

Minimising waste 

AI is also helping businesses to drive profit, minimising wasted resources, and identifying potential disruptions. By tracking levels of supply and demand, AI can automatically identify challenges such as stock shortages, delivery-route disruptions, or a heightened demand for a particular product. More impressively, however, they are also capable of suggesting solutions to these problems – from the fastest delivery route that avoids traffic, to diverting stock to a new warehouse. Such planning and preparation help businesses to avoid disruptions which costs valuable time, money, and resources. 

According to Forbes Advisor, 56% of businesses are already using AI for customer service, and 47% for digital personal assistance. If organisations want to keep up with their cutting edge-competitors, AI tools are quickly becoming a must-have for their inventory. 

For SMBs looking to stay afloat in this competitive landscape of AI innovation, getting the most out of their technological investment is crucial. 

Laying down the foundations

Adopting AI isn’t as straightforward as ‘plug and play’ and SMBs shouldn’t underestimate the investment these tools require. Whilst many of the applications may be easy to use, it’s important that business leaders take time to fully understand the technology and its potential uses. Otherwise, they risk missing some major benefits and not getting the most from their investment, particularly as they scale out. 

Acknowledging the potential risks and challenges of implementing new AI tools can help organisations prepare solutions and ensure that their business is equipped to manage the modern technology. This can help businesses to avoid costly mistakes and hit the ground running with their innovation efforts. 

SMB leaders looking to implement AI first need to ask the following:

What can AI do for me? 

Are day-to-day administration tasks your biggest sticking points? Or are you looking to provide customer service like no-other? Identifying how AI might be of most use for your business can help you to make the most effective investments. It’s also worth considering the tools and applications you already have, and how AI might enhance these. Many companies already use Microsoft Office, for instance, which Microsoft Copilot can seamlessly slot into, making for a much smoother rollout. 

Can my business manage its data? 

AI is powered by data, so having sufficient data-management and storage processes in place is necessary. Before investing in AI, businesses might benefit from first looking at managed data platforms and services. This is crucial for providing the scalability, security and flexibility needed to embrace innovation in a responsible and effective way. 

What about regulation?

The use and development of AI are becoming increasingly regulated, with legislation such as the EU AI Act providing stringent, risk-based guidance on its adoption. Keeping up with the latest rules and legislative changes is vital. Not only will this help your business to maintain compliance, but it will also help to maintain trust with customers and employees alike, whose data might be stored and processed by AI. Reputational damage caused by a data breach is a tough blow even for big businesses, so organisations would be wise to avoid it where possible. 

Embracing innovation

This new age of AI is exciting; it holds great transformative potential. We’ve already seen the development of accessible, affordable tools, such as Microsoft Copilot, opening a world of new innovative potential to businesses of all sizes. Those that don’t dip their toes in the AI pool risk getting left behind. 

The question smaller businesses ask themselves can no longer be about whether AI is right for them; instead, it should be about how they can best access its benefits within the parameters of their budget. 

By thoroughly preparing and taking time to understand the full process of AI adoption, SMBs can make sure that their digital transformation efforts are a success. In today’s world, this is the best way to remain fiercely competitive in a continuously evolving landscape. 

Anthony Coates Smith, Managing Director of Insite Energy, takes a look at developments in the data-driven heating systems helping our cities reach net zero.

Heat networks – communal heating systems fed by a single, often locally generated, renewable, heat source – are a crucial component of government strategy to clean up the UK’s energy supply. With strong potential to reduce carbon emissions in urban areas, they’re fast becoming the norm in modern residential and commercial developments. In fact, they’re expected* to meet up to 43% of the country’s residential heat demand by our 2050 net-zero deadline – a meteoric rise from just 2% in 2018.

The key word here, though, is ‘potential’. Compared to other European countries, advanced heat network technologies are still vastly underused and widely unfamiliar in the UK. The market has not yet had time to accumulate the experience and expertise needed to design, operate and maintain these highly complex systems at their optimum. Consequently, most are running at just 35-45% efficiency** leaving the entire sector in a precarious position.

It can be helpful to think of a heat network as a bit like a luxury car. It’s a high-value, expertly engineered asset that needs skilful and consistent servicing to protect its value and ensure its reliability and longevity. If you compare a modern vehicle to a 1980s equivalent, the technology is very different. It’s much greener and more efficient, with a far greater emphasis on digitalisation and data. 

UK catch-up

The same is true of heat networks, but the UK industry still has a way to go to take full advantage of these developments. We’re on a mission to change that. We work with heat network operators to help them use data and digital technologies to reduce costs and carbon emissions, enhance efficiency and reliability, change consumer behaviours, boost engagement and improve customer experience. 

One way we do this is by developing and introducing new technologies and services into the UK heat network market that already exist in other countries or other industries but have no precedent here. 

A notable example is KURVE. The first web-app for heat network residents to monitor their energy consumption and pay their bills, KURVE brings the same levels of customer experience and functionality that banking customers, for example, have benefitted from for years. 

Giving people real-time information that empowers them to manage their energy use can significantly reduce consumption. In households using KURVE, it drops by around 24% on average. Furthermore, the data analysis KURVE has enabled has informed and improved industry best practice around sustainability and user experience.

The power of pricing

Another recent innovation was our introduction of motivational tariffs to the UK heat network sector in 2023. This is a form of variable pricing providing financial incentives to encourage energy-saving behaviours. It directly tackles the ‘What’s in it for me?’ problem inherent in communal heating systems, where customers’ heating bills are at least as dependent on their neighbours’ actions as their own. 

Motivational tariffs have been used to great effect in Denmark, where 64% of homes are on heat networks. In the UK, results have included lower bills for 81% of residents and a seven-fold increase in uptake of equipment-servicing visits.

A third example is the use of digital twinning to tackle poor operational performance. A heat network is a vast web of interconnected components; any intervention will have impacts across the entire system that are not always predictable. Creating an accurate virtual model of its hydronic design enables you to see if it’s as good as it can be – and if not, why not. You can then try out different options to obtain the best results – without the expense, risk or disruption of real-world alterations. 

Over the past five years, digital twins have, among other things, helped a member of our team optimise the heat network supplying the world-famous green houses at Kew Gardens and prevent a huge engineering undertaking that would have had little impact at a 190-unit London apartment building. Despite the evident benefits however, we’re still alone in the UK in proselytising and practising digital twinning for these types of purposes.

Mainstream

I’m glad to say that some data-driven technologies have been widely adopted to good effect. Smart meters, in-home devices and pay-as-you-go billing systems are now common, giving residents accurate real-time information and better control over their energy use. Smart technology is also deployed in plant rooms and across networks to monitor and respond to changes in demand and environmental conditions. 

Heat network operators are increasingly waking up to the importance of continuous and meticulous monitoring of performance data to spot faults and inefficiencies quickly and tailor heat supply to minimise network losses. This can happen remotely using cloud-based services, which can also help to diagnose and even fix some issues, keeping repair costs low.

What’s next?

An area where there’s likely to be further innovation in the near future is big data visualisation to make performance monitoring easier and more effective. As many heat network operators are organisations like housing associations and local authorities, with numerous competing concerns vying for their attention, anything that can translate complex technical information into simple graphics is welcome. And linked to this will be further enhancements in performance reporting and visualisation for customers.

We can also expect to see greater use of integrated heat source optimisation, whereby dynamic monitoring and switching are used to select the lowest cost/carbon heat source at any given time.

One thing we don’t anticipate any time soon, however, is AI chat bots replacing human customer-service interactions. While there’s a place for AI in heat network customer care, it’s more at the smart information services end of the spectrum. The recent energy and cost-of-living crises have underlined the importance of the human touch when it comes to something as fundamental as heating your home. 

*Source: 2018 UK Market Report from The Association for Decentralised Energy** Source: The Heat Trust

In an effort to decrease its reliance on overseas hyperscalers, Europe has set its sights on data independence. 

This was a challenging issue from the get-go but has been further complicated by the rise of AI. Countries want to capitalise on its potential but, to do that, they need access to the world’s best minds and technology to collaborate and develop the groundbreaking AI solutions that will have the desired impact. Therein is the challenge. How to create the technical landscape to enable AI to thrive whilst not compromising sovereignty. 

Governments and the AI goldrush

Let’s not beat around the bush. This is something Europe needs to get ‘right first time’ because of the speed at which AI is moving. Nvidia CEO Jensen Huang recently underlined the importance of Sovereign AI. Huang stressed the criticality of countries retaining control over their AI infrastructure to preserve their cultural identity. 

It’s why it is an issue at the top of every government agenda. For instance, in the UK, Baroness Stowell of Beeston, Chairman of the House of Lords Communications and Digital Committee, recently said, “We must avoid the UK missing out on a potential AI goldrush”. It’s also why countries like the Netherlands have developed an open LLM called GPT-NL. Nations want to build AI with the goal of promoting their nation’s values and interests. The Netherlands is also jointly promoting a European sovereign AI plan to become a world leader in AI. There are many other instances of European countries doing or saying something similar.

A new class of accelerated, AI-enabled infrastructure

The WEF has a well-publicised list of seven pillars needed to unlock the capabilities of AI – talent, infrastructure, operating environment, research, development, government strategy and commercial. However, this framework is as impractical as it is admirable. For such a rapidly moving issue, governments need something more pragmatic. They need a simple directive focused at the technological level to make the dream of AI sovereignty a reality. 

This will involve a new class of accelerated, AI-enabled infrastructure that feeds enormous amounts of data to incredibly powerful compute engines. Directed by sophisticated software, this new infrastructure could create a neural network capable of learning faster and applying information faster than ever before. So, how best to bring this to life?

A fundamental element of openness

For a start, for governments to achieve AI sovereignty, they must think about a solid, secure and compliant data foundation. It is imperative that the data they are working with has been subject to the highest levels of hygiene. Beyond this, they need the capabilities to scale. AI involves training and retraining data while regulation is also likely to evolve in the coming years. Therefore, without the ability to scale, innovation will be stifled. That means it is imperative to have an infrastructure with a fundamental element of openness on several levels.

Open data models 

Achieving sovereignty for each state will be impossible without collaboration and alliances. It will simply be too expensive and some countries do not have pockets as deep as hyperscalers. This means a strategy for Europe must not only have open data models that countries can share, but also involve clever ways of using the available funding. For instance, instead of creating a fund that many disconnected private companies can access, invest it in building a company that is specifically focused on one aspect of AI sovereignty that can be distributed Europe-wide for nations to adapt.

Open data formats 

When it comes to sovereignty, it’s not as arbitrary as having open or closed data. Some data, like national security, is sensitive and should never be exposed to anybody outside a nation’s borders. However, there are other types of data that could be open and accessible for everyone which would cost-effectively allow nations to train models within with that data and create appropriate sovereign AI products and protocols as a result. 

Open data verification 

One of the challenges with AI is data provenance. Without standardised and established methods for verifying where data came from, there are no guarantees that available data is what it claims to be. There is no reason that a European-wide standard for data provenance cannot be agreed upon in much the same way as the sourced footnotes in Wikipedia. 

Open technology

In the context of sovereignty, this might seem counterintuitive but it has been done successfully and recently with the Covid tracking app. The software ensured that personal data was protected at a national and individual level but that the required information was shared for the greater good. This should be the model for achieving AI sovereignty in Europe.

Transformative impact of open source

This is where open source (OSS) technology can be transformative. For a start, it’s the most cost-effective approach. What’s more, realistically, it’s the only way nations will be able to build the programmes they need. Beyond the money, one of the founding principles of OSS was that it was open to study and utilise with no restrictions or discrimination of use. It can be adopted and built upon in a way that suits nations while not compromising on security or data sovereignty. This ability to understand and modify software, hardware and systems independently and free from corporate or top-down control gives countries the ability to run things on their own terms. 

Finally, and perhaps most importantly, it can scale. Countries can always be on the latest version without depending on a foreign country or private enterprise for licensing requirements. It allows countries to benefit from a local model but, at the same time, have boundaries on the data.

A debate we don’t want to continue

When it comes to AI sovereignty, openness could be considered antithetical. However, the reality is that sovereignty will not be achieved without it. If nations persist in being closed books, we’ll still be having this debate in years to come – by which point it may be too late.

The fact is, nations need AI to be open so they can build on it, improve it, and ensure privacy. Surely that is what being sovereign is all about?

The integration of AI into business operations is no longer a question of if, but how. Companies across industries are increasingly recognising the potential of AI to deliver significant business benefits. Applying AI to emissions data can unlock valuable insights that help organisations reduce their environmental impact and capitalise on emerging opportunities in the sustainability space.

Navigating the Challenges of Emissions Data

Organisations face two primary challenges when managing emissions data. The first is regulatory compliance. Governments worldwide are implementing stricter emissions reporting requirements, and businesses must demonstrate ongoing reductions. 

To meet these demands, companies need a clear understanding of their current emissions footprint and the areas within their operations or supply chain where changes can lead to reductions. Moreover, they must implement these changes and track their progress over time.

The second challenge involves identifying business opportunities linked to emissions data. For example, the US’ Inflation Reduction Act offers investment credits for initiatives like carbon sequestration and storage, presenting significant financial incentives for companies that can efficiently manage and analyse their emissions data.

AI plays a pivotal role in addressing both challenges. By processing vast emissions datasets, AI can pinpoint areas within a company’s operations that offer the greatest potential for emissions reduction. It can also identify investment opportunities that align with sustainability initiatives. However, the effectiveness of AI depends on the quality and consistency of the emissions data.

The Role of Data Consistency in AI-Driven Insights

Before AI can be applied effectively to emissions data, the data must be well-organised and standardised. Consistency is critical, not only in the data itself but also in the associated metadata—such as units of measurement, emissions calculation formulas, and categories of emissions components. Additionally, emissions data must align with the organisational structure, covering factors like location, facility, equipment, and product life cycles.

Inconsistent data hinders the performance of AI models, leading to unreliable results. As Robert Seltzer highlights in his article Ensuring Data Consistency and Standardisation in AI Systems, overcoming challenges like diverse data sources, inconsistent data models, and a lack of standardisation protocols is essential for improving AI performance. When applied to emissions data, these challenges become even more pronounced. While greenhouse gas (GHG) data standards exist, the absence of a ubiquitous data model means that businesses often struggle with inconsistent data formats, especially when managing scope 3 emissions data from suppliers.

Implementing Standardised Data Models

One solution is the adoption of standardised data models, such as the Open Footprint Data Model. 

This model ensures consistency in data naming, units of measurement, and relationships between data elements, all of which are essential for applying AI effectively to emissions data. By standardising data, companies can eliminate the need for manual conversion processes, accelerating the time to value for AI-driven insights.

Use Cases for AI in Emissions Data

Consider the example of a large multinational corporation with an extensive supply chain. This company wants to use AI to analyse the emissions profiles of its suppliers and identify which suppliers are effectively reducing emissions over time. 

For AI to deliver meaningful insights, the emissions data from each supplier must be consistent in terms of definitions, metadata, and units of measure. Without a standardised approach, companies relying on spreadsheets would face labour-intensive data conversion efforts before AI could even be applied.

In another scenario, a company seeks to evaluate its scope 1 and 2 emissions across various business units, identifying areas where capital investments could yield the greatest emissions reductions. 

Here, it’s essential that emissions data from different parts of the business be comparable, requiring consistent data definitions, units of measure, and calculation methods. As with the previous example, the use of a standard data model simplifies this process, making the data AI-ready and reducing the need for manual intervention.

The Business Case for a Standard Emissions Data Model

Adopting a standard emissions data model offers numerous advantages. Not only does it reduce the complexity of collecting and managing data from across an organisation and its supply chain, but it also facilitates the application of AI, enabling advanced analytics that drive emissions reductions and uncover new business opportunities. 

For companies seeking to maximise the value of their emissions data, standardisation is a critical first step.

By embracing a standardised data framework, businesses can overcome the barriers that prevent AI from unlocking the full potential of their emissions data, ultimately leading to more sustainable practices and improved financial outcomes.

Discussions around future employability tend to highlight the unique qualities that we, as humans, value. While we might pride ourselves on our emotional intelligence, communication skills and creativity, it leaves a set of skills that would have our secondary school careers advisors directing us all off to retrain in nursing and the creative arts. And, quite honestly, if I have a tricky email to send, Chat GPT does a much better job at writing with immense tact than I do.

Fortunately for us all, these simplifications of such a complex issue overlook some reassuring limitations built into the Transformers architecture, the technology that the latest and most impressive generation of AI is built on. 

The limits of modern AI

These tools have learnt to be literate in the most basic sense. They can predict the next, most logical, token that will please their human audience. The human audience can then connect that representation to something in the real world. There is nothing in the transformers architecture to help answer questions like ‘Where am I right now?’ or ‘What is happening around me?’ 

In business these are often crucial questions. The architecture can’t just be tweaked to add that as an upgrade. Unless someone has already built an alternative architecture in secret somewhere in Silicon Valley, we won’t see a machine that combines Chat GPT with contextual understanding any time soon. 

Where transformers have been revolutionary, it tends to be areas where humans had almost given up the job. Medical research, for example, is a terrifically expensive and failure-ridden process. But using a well-trained transformer to sift through millions of potential substances to identify candidates for human development and testing is making success a more familiar sensation for our medical researchers. But that kind of success can’t be replicated everywhere.

Joining it all up

We, of course, have some wonderful examples of technologies that can actually answer questions like ‘Where am I and what’s going?’ Your satnav, for one, has some idea where you are and of some hazards ahead. More traditional neural networks can look at images of construction sites and spot risk hazards before they become an accident. Machines can look at medical scans and see if cancer is or is not present. 

But these machines are highly specialised. The same AI can’t spot hazards around my home, or in a school. The machine that can spot bowel cancer can’t be used to detect lung cancer. This lack of interaction between highly specialised algorithms means that, for now, AI still needs a human running the show. They must choose which machine to use, and whether to override the suggestions that the machine makes.

AI: Confidently wrong

And that is the other crucial point. Many of the algorithms that are being embedded into our workplace have very poor understanding of their own capabilities. They’re like the teenager who thinks they’re invincible because they haven’t experienced failure and disappointment often enough yet. 

If you train a machine to recognise road signs, it will function very well at recognising clean, clear road signs. We would expect it to struggle more with ‘edge’ cases. Images of dirty, mud-splattered road signs taken at night during a storm, for example, trip up AI where humans succeed. But what if you show it something completely different, like images of foods? 

Unless it has also been taught that images of food are not road signs and need a completely different classification, the machine may well look at a hamburger and come to the conclusion that – of all the labels it can apply – it most clearly represents a stop sign. The machine might make that choice with great confidence – a circle and a line across the middle – it’s obviously not a give way sign! So human oversight to be able to say, ‘Silly machine, that’s a hamburger!’ is essential. 

What does this mean for the next 10 years of your career?

It does not mean the end of your career, unless you are in a very small and unfortunate category of professions. But it does mean that the most complex decisions you have to take today are soon going to become the norm. The ability to make consistent, adaptable, high quality decisions is vital to helping your career to flourish. 

Fortunately for our careers, the world is unlikely to run out of problems to solve any time soon. 

With complex chains of dependencies and huge volatility in world markets, it’s not enough to evolve your intelligence to make more rational decisions (although that will always help – we are, by default, highly emotional decision makers). 

To make great decisions, you need to know what you can’t compute, and what the machines can’t compute. There will be times when external insights from data can support you in decision making. But there will also be intermediaries to coordinate, errors to identify, and competing views on solutions to weigh up. 

All machine intelligence requires compromise, and fortunately, that limitation leaves space for us, but only if we train ourselves to work in this new professional environment. At the Institute of Analytics, we work with professionals to support them in this journey. 

Dr Clare Walsh is a leading academic in the world of  data and AI, advising governments worldwide on ethical AI strategies. The IoA is a global, not-for-profit professional body for analytics and data professionals. It promotes the ethical use of data-driven decision making and offers membership services to individuals and businesses, helping them stay at the cutting edge of analytics and AI technology.

It’s getting to the time of year when priorities suddenly come into sharp focus. Just a few months ago, 2024 was fresh and getting started. Now, the days and weeks are being ticked off the calendar at breakneck speed, and with 2025 within touching distance, many CIOs will be under pressure to deliver before the year is out. 

This isn’t about juggling one or two priorities. Most CIOs are stretched across multiple projects on top of keeping their organisations’ IT systems on track; from delivering large digital transformation projects and fending off cyber attacks, to introducing AI and other innovative tech.

So, where should CIOs put their focus in the last months of 2024, when they face competing priorities and time is tight? How do they strike the right balance between innovation and overall performance? 

We’ve asked a panel of experts to share what they think will make the most impact, when it comes to data.

Get your data in order

Building a strong foundation for current and future projects is a great place to start, according to our specialists. First stop, managing data. Specifically data quality.

“Without the right, accurate data, the rest of your initiatives will be challenging: whether that’s a complex migration, AI innovation or simply operating business as usual,” Syniti MD and SVP EMEA Chris Gorton explains. “Start by getting to know your data, understanding the data that’s business critical and linked to your organisational objectives. Next, set meaningful objectives around accuracy and availability, track your progress and be ready to adjust your approach if needed. Then introduce robust governance your organisation can follow to make sure your data quality remains on track. 

“By putting data first over the next few months, you’ll be in a great position to move forward with those big projects in 2025.”

As well as giving a good base to build from, getting to grips with data governance can also help to protect valuable data. 

Keepit CISO Kim Larsen points out: “When organisations don’t have a clear understanding and mapping of their data and its importance, they cannot protect it or determine which technologies to implement, and therefore preserve that data and determine who has access to it.

“When disaster strikes and they lose access to their data, whether because of cyberattacks, human error or system outages, it’s too late to identify and prioritise which data sets they need to recover to ensure business continuity. Good data governance equals control. In a constantly evolving cyber threat landscape, control is essential.”

Understand the infrastructure you need behind the scenes

Once CIOs are confident of their data quality, infrastructure may well be the next focus: particularly if AI, Machine Learning or other innovative technologies are on the cards for next year. Understanding the infrastructure needed for optimum performance is key, otherwise new tools may fail to deliver the results they promise.

Xinnor CRO Davide Villa explains: “As CIOs implement innovative solutions to drive their businesses forward, it’s crucial to consider the foundation that supports them. Modern workloads like AI, Machine Learning, and Big Data analytics all require rapid data access. In recent years, fast storage has become an integral part of IT strategy, with technologies like NVMe SSDs emerging as powerful tools for high-performance storage.

“However, it’s important to think holistically about how these technologies integrate with existing infrastructures and data protection methods. As you plan for the future, take time to assess your storage needs and explore various solutions. Determine whether traditional storage solutions best suit your workload or if more modern approaches, such as software-based versions of RAID, could enhance flexibility and performance. The goal is to create an infrastructure that not only meets your current demands efficiently but also remains adaptable to future requirements, ensuring your systems can handle evolving workloads’ speed and capacity needs while optimising resource utilisation.”

Protect against cyber attacks…

With threats from AI-powered cyber crime and ransomware increasing, data protection is high on our experts’ priorities.

As a first step, Scality CMO Paul Speciale says “CIOs should assess their existing storage backup solutions to make sure they are truly immutable to provide a baseline of defence against ransomware that threatens to overwrite or delete data. Not all so-called immutable storage is actually safe at all times, so inherently immutable object storage is a must-have.

“Then look beyond immutable storage to stop exfiltration attacks. Mitigating the threat of data exfiltration requires a multi-layered approach for a more comprehensive standard of end-to-end cyber resilience. This builds safeguards at every level of the system – from API to architecture – and closes the door on as many threat vectors as possible.”

Piql founder and MD, Rune Bjerkestrand, agrees: “We rely on trusted digital solutions in almost every aspect of our lives, and business is no exception. And although this offers us many opportunities to innovate, it also makes us vulnerable. Whether those threats are physical, from climate change, terrorism, and war, or virtual, think cyber attack, data manipulation and ransomware, CIOs need to ensure guaranteed, continuous access to authentic data.

“As the year comes to an end, prioritise your critical data and make sure you have the right protection in place to guarantee access to it.”

Understanding the wider cyber crime landscape can also help to identify the most vulnerable parts of an infrastructure, says iTernity CEO Ralf Steinemann. “In these next few months, prioritise business continuity. Strengthen your ransomware protection and focus on the security of your backup data. Given the increasing sophistication and frequency of ransomware attacks, which often target backups, look for solutions that ensure data remains unaltered and recoverable. And consider how you’ll further enhance security by minimising vulnerabilities and reducing the risk of human error.”

Remember edge data

Central storage and infrastructure is a high priority for CIOs. But with the majority of data often created, managed and stored at the edge, it’s incredibly important to get to grips with this critical data.

StorMagic CTO Julian Chesterfield explains: “Often businesses do not apply the same rigorous process for providing high availability and redundancy at the edge as they do in the core datacentre or in the cloud. Plus, with a larger distributed edge infrastructure comes a larger attack surface and increased vulnerabilities. CIOs need to think about how they mitigate that risk and how they deploy trusted and secure infrastructure at their edge locations without compromising integrity of overall IT services.”

Think long term

With all these competing challenges, CIOs must make sure whatever they prioritise supports the wider data strategy, so that the work put in now has long-term benefits, say Pure Storage Field CTO EMEA Patrick Smith. 

“CIO focus should be on a long term strategy to meet these multiple pressures. Don’t fall into the trap of listening to hype and making decisions based on FOMO,” he warns. “Given the uncertainty associated with some new initiatives, consuming infrastructure through an as-a-Service model provides a flexible way to approach these goals. The ability to scale up and down as needed, only pay for what’s being used, and have guarantees baked into the contract should be an appealing proposition.”

Where will you focus?

As we enter the final stretch of 2024, it’s crucial to prioritise and take action. With the right strategies in place focusing on data quality, governance, infrastructure, and security, CIOs will be set up to meet current demands, and build a solid foundation for their organisations in 2025 and beyond. 

Don’t wait for the pressures to mount. The experts agree: start prioritising now, and get ready to thrive in the year ahead.

Interest in implementing AI is truly global and industry-agnostic. However, few companies have established the foundational building blocks that enable AI to generate value at scale. While each organisation and industry will have their own specific challenges that may impact AI adoption, there are four common barriers that all companies tend to encounter: People, Control of AI models, Quality, and Cost. To implement AI successfully and ensure long-term value creation, it’s critical that organisations take steps to address these challenges.

Accessible upskilling 

At the forefront of these challenges is the impending AI skills gap. The speed at which the technology has developed demands attention, with executives estimating that 40% of their workforce will need to re-skill in the next three years as a result of implementing AI – outlying that this is a challenge that requires immediate attention.

To tackle this hurdle, organisations must provide training that is relevant to their needs, while also establishing a culture of continuous learning in their workforce. As the technology continues to evolve and new iterations of tools are introduced, it’s vital that workforces stay up to date on their skills.

Equally important is democratising AI upskilling across the entire organisation – not just focusing on tech roles. Everyone within an organisation, from HR and administrative roles to analysts and data scientists, can benefit from using AI. It’s up to the organisation to ensure learning materials and upskilling initiatives are as widely accessible as possible. However, democratising access to AI shouldn’t be seen as a radical move that instantly prepares a workforce to use AI. Instead, it’s crucial to establish not just what is rolled out, but how this will be done. Organisations should consider their level of AI maturity, making strategic choices about which teams have the right skills for AI and where the greatest need lies. 

Consider AI models

As organisations embrace AI, protecting data and intellectual property becomes paramount. One effective strategy is to shift focus from larger, generic models (LLMs) to smaller, customised language models and move toward agentic or compound AI systems. These purpose-built models offer numerous advantages, including improved accuracy, relevance to specific business needs, and better alignment with industry-specific requirements.

Custom-built models also address efficiency concerns. Training a generalised LLM requires significant resources, including expensive Graphics Processing Units (GPUs). Smaller models require fewer GPUs for training and inference, benefiting businesses aiming to keep costs and energy consumption low.

When building these customised models, organisations should use an open, unified foundation for all their data and governance. A data intelligence platform ensures the quality, accuracy, and accessibility of the data behind language models. This approach democratises data access, enabling employees across the enterprise to query corporate data using natural language, freeing up in-house experts to focus on higher-level, innovative tasks.

The importance of data quality 

Data quality forms the foundation of successful AI implementation. As organisations rush to adopt AI, they must recognise that data serves as the fuel for these systems, directly impacting their accuracy, reliability, and trustworthiness. By leveraging high-quality, organisation-specific data to train smaller, customised models, companies ensure AI outputs are contextually relevant and aligned with their unique needs. This approach not only enhances security and regulatory compliance but also allows for confident AI experimentation while maintaining robust data governance.

Implementing AI hastily without proper data quality assurance can lead to significant challenges. AI hallucinations – instances where models generate false or misleading information – pose a real threat to businesses, potentially resulting in legal issues, reputational damage, or loss of trust. 

By prioritising data quality, organisations can mitigate risks associated with AI adoption while maximising its potential benefits. This approach not only ensures more reliable AI outputs but also builds trust in AI systems among employees, stakeholders, and customers alike, paving the way for successful long-term AI integration.

Managing expenses in AI deployment

For C-suite executives under pressure to reduce spending, data architectures are a key area to examine. While a recent survey found that Generative AI has skyrocketed to the #2 priority for enterprise tech buyers, and 84% of CIOs plan to increase AI/ML budgets, 92% noted they don’t have a budget increase over 10%. This indicates that executives need to plan strategically about how to integrate AI while remaining within cost constraints.

Legacy architectures like data lakes and data warehouses can be cumbersome to operate, leading to information silos and inaccurate, duplicated datasets, ultimately impacting businesses’ bottom lines. While migrating to a scalable data architecture, such as a data lakehouse, comes with an initial cost, it’s an investment in the future. Lakehouses are easier to operate, saving crucial time, and are open platforms, freeing organisations from vendor lock-in. They also simplify the skills needed by data teams as they rationalise their data architecture.

With the right architecture underpinning an AI strategy, organisations should also consider data intelligence platforms to leverage data and AI by being tailored to its specific needs and industry jargon, resulting in more accurate responses. This customisation allows users at all levels to effectively navigate and analyse their enterprise’s data.

Consider the costs, pump the brakes, and take a holistic approach

Before investing in any AI systems, businesses should consider the costs of the data platform on which they will perform their AI use cases. Cloud-based enterprise data platforms are not a one-off expense but form part of a business’ ongoing operational expenditure. The total cost of ownership (TCO) includes various regular costs, such as cloud computing, unplanned downtime, training, and maintenance.

Mitigating these costs isn’t about putting the brakes on AI investment, but rather consolidating and standardising AI systems into one enterprise data platform. This approach brings AI models closer to the data that trains and drives them, removing overheads from operating across multiple systems and platforms.

As organisations navigate the complexities of AI adoption, addressing these four main barriers is crucial. By taking a holistic approach that focuses on upskilling, data governance, customisation, and cost management, companies will be better placed for successful AI integration.  

With the launch of the Labour Government’s Autumn Budget, Sir Kier Starmer’s government and Chancellor Rachel Reeves seem determined to convince Labour voters that the adults are back in charge of the UK’s finances, and convince conservatives that nothing all that fundamental will change. Popular policies like renationalising infrastructure are absent. Some commenters worry that Reeves’ £40 billion tax increase will affect workers in the form of lower wages and slimmer pay rises. 

Nevertheless, tech industry experts have hailed more borrowing, investment, and productivity savings targets across government departments as positive signs for the UK economy. In the wake of the budget’s release, we heard from three leaders in the UK tech sector about their expectations and hopes for the future. 

Growth driven by AI 

Damian Stirrett, Group Vice President & General Manager UK & Ireland at ServiceNow 

“As expected, growth and investment is the underlying message behind the UK Government’s Autumn Budget. When we talk about economic growth, we cannot leave technology out of the equation. We are at an interesting point in time for the UK, where business leaders recognise the great potential of technology as a growth driver leading to impactful business transformation.   

AI is, and will increasingly be, one of the biggest technological drivers behind economic growth in the UK. In fact, recent research from ServiceNow, has found that while the UK’s AI-powered business transformation is in its early days, British businesses are among Europe’s leaders when it comes to AI optimism and maturity, with 85% of those planning to increase investment in AI in the next year. It is clear that appetite for AI continues to grow- from manufacturing to healthcare, and education. Furthermore, with the government setting a 2% productivity savings target for government departments, AI has the potential to play a significant role here, not only by boosting productivity, but driving innovation, reducing operational costs, as well as creating new job opportunities.   

To remain competitive as a country, we must not forget to also invest in education, upskilling initiatives, and partnerships between the public and private sectors, fostering AI innovation to drive transformative change for all.” 

Investing in the industries of the future

By James Hall, Vice President and Country Manager UK&I at Snowflake

“Given the Autumn budget’s focus on investing in industries of the future, AI must be at the forefront of this innovation. This follows the new AI Opportunities Action Plan earlier this year, looking to identify ways to accelerate the use of AI to better people’s lives by improving services and developing new products. Yet, to truly capitalise on AI’s potential, the UK Government must prioritise investments in data infrastructure.

AI systems are only as powerful as the data they’re trained on; making high-quality, accessible data essential for innovation. Robust data-sharing frameworks and platforms enable more accurate AI insights and drive efficiency, which will help the UK remain globally competitive. With the right resources, the UK can lead in offering responsible and effective AI applications. This will benefit both public services and the wider economy, helping to fuel smart industries and meet the growth goals set out by the Chancellor.” 

Growth, stability, and a careful, considered approach 

By Rupal Karia, VP & Country Leader UK&I at Celonis

“Hearing the UK Government’s autumn budget, it’s clear that growth and stability are the biggest messages. With the Chancellor outlining a 2% productivity savings target for government departments, it is crucial the public sector takes heed of the role of technology which cannot be understated as we look to the future. Artificial intelligence is being heralded by businesses, across multiple sectors, as a game-changing phenomenon. Yet for all of the hype, UK businesses must take a step back and consider how to make the most of their AI investments to maximise ROI. 

The UK must complement investments in AI with a strong commitment to process intelligence technology. AI holds transformative potential for both the public and private sectors, but without the relevant context being provided by process intelligence, organisations risk failing to achieve ROI. Process intelligence empowers businesses with full visibility into how internal processes are operating, pinpointing where there are bottlenecks, and then remediates these issues. It is the connective tissue that gives organisations the insight and context they need to drive impactful AI use cases which will help businesses achieve return on AI investment. 

Celonis’ research reveals that UK business leaders believe that getting support with AI implementation would be more important for their businesses than reducing red tape or cutting business rates. This is a clear guideline for the UK government to consider when looking to fuel growth.” 

Prior to the mainstream AI revolution, which started with the public launch of ChatGPT, organisations were already embedding AI in one form or another into security controls for some time. Historically, security product developers have favoured using Machine Learning (ML) in rheir products, dating back to the millennium when intrusion detection systems began to use complex models to identify unusual network traffic.  

Machine learning and security 

Since then, developers have employed ML in many categories of security products, as it excels in organising large data sets. 

If you show a machine learning model a million pictures of a dog, followed by a million pictures of a cat, it can determine with pretty good accuracy whether a new, unseen image is of a dog or a cat. 

This works the same way with ‘legitimate’ and ‘malicious’ data. Today, it would be unusual to find an antivirus product for sale that does not incorporate ML functionality. It works well, and it isn’t easily fooled by slight changes to a virus, for example. This is important with the speed of change in today’s threat landscape. 

LLM security applications 

ChatGPT is a type of Artificial Intelligence that falls under the category of a ‘Large Language Model’ (LLM). LLMs are relatively new to the security market, and there is a rush from vendors to jump on the bandwagon and incorporate this type of AI into their products. 

Two areas of greatest value so far include the ability to summarise complex technical information – such as ingesting the technical details about a security incident and describing it – and how to remediate it, in an easy-to-understand way. 

The reverse is also true, many complex security products which previously required the administrator to learn a complex scripting language to interact with it, can now ask it simple questions in their native language. 

The LLM will ‘translate’ these queries into the specific syntax required by the tool. 

This is enabling organisations to get more value from their junior team members, and reducing the time-to-value for new employees. We’re likely to see some of the ‘heavy lifting’ of repetitive tasks offloaded to AI models.  

LLM AI integration requires organisations to keep both eyes open 

When integrating AI security tools, businesses must establish policies and training to ensure staff can leverage these tools effectively. Protecting sensitive training data and understanding privacy policies are crucial to mitigating data privacy risks. 

Additionally, businesses should keep informed about the latest developments and updates so they can ensure continuous improvement of their AI tools. This approach ensures AI tools augment security while aligning with ethical standards and organisational policies, maintaining the balance between technology and human expertise.  

Finally, organisations must remain vigilant when it comes to developments in regulation. For instance, the EU Artificial Intelligence Act, which will start to take effect over the next 12 months, requires organisations to ensure that their AI systems comply with stringent requirements regarding safety, transparency, and accountability. 

This includes conducting risk assessments, ensuring data quality and robustness, providing clear and understandable information to users, and establishing mechanisms for human oversight and control. Businesses must use documentation AI system activity logging Prior to the mainstream AI revolution, which started with the public launch of ChatGPT, AI in some form had been embedded into security controls for some time. Historically, Machine Learning (ML) has been the category of AI used in security products, dating back to the millennium when intrusion detection systems began to use complex models to identify unusual network traffic.  

Machine learning and security 

Since then, organisations have used ML in many categories of security products, as it excels in organising large data sets. 

If you show a machine learning model a million pictures of a dog, followed by a million pictures of a cat, it can determine with pretty good accuracy whether a new, unseen image is of a dog or a cat. 

This works the same way with ‘legitimate’ and ‘malicious’ data. Today, it would be unusual to find an antivirus product for sale that does not incorporate ML functionality. It works well, and it isn’t easily fooled by slight changes to a virus, for example. This is important with the speed of change in today’s threat landscape. 

LLM security applications 

ChatGPT is a type of Artificial Intelligence that falls under the category of a ‘Large Language Model’ (LLM). LLMs are relatively new to the security market, and there is a rush from vendors to jump on the bandwagon and incorporate this type of AI into their products. 

Two areas of greatest value so far include the ability to summarise complex technical information – such as ingesting the technical details about a security incident and describing it – and how to remediate it, in an easy-to-understand way. 

The reverse is also true, many complex security products which previously required the administrator to learn a complex scripting language to interact with it, can now ask it simple questions in their native language. 

The LLM will ‘translate’ these queries into the specific syntax required by the tool. 

This is enabling organisations to get more value from their junior team members, and reducing the time-to-value for new employees. We’re likely to see companies offload some of the ‘heavy lifting’ of repetitive tasks to AI models. This in turn will free up more time for humans to use their expertise for more complex and interesting tasks that aid staff retention.

These models are also prone to ‘hallucinate’. Whn this happens, AI models make up information that is completely incorrect. Because of this, it’s important not to become overly reliant on AI – using it as an assistant rather than a replacement for expertise, and to avoid becoming exclusively dependent on it.  

LLM AI integration requires organisations to keep both eyes open 

When integrating AI security tools, businesses must establish policies and training to ensure staff can leverage these tools effectively. Protecting sensitive training data and understanding privacy policies are crucial to mitigating data privacy risks. 

Additionally, businesses should keep informed about the latest developments and updates so they can ensure continuous improvement of their AI tools. This approach ensures AI tools augment security while aligning with ethical standards and organisational policies, maintaining the balance between technology and human expertise.  

Finally, organisations must remain vigilant when it comes to developments in regulation. For instance, the EU Artificial Intelligence Act, which will start to take effect over the next 12 months, requires organisations to ensure that their AI systems comply with stringent requirements regarding safety, transparency, and accountability. 

This includes conducting risk assessments, ensuring data quality and robustness, providing clear and understandable information to users, and establishing mechanisms for human oversight and control. Businesses must also maintain thorough documentation and logging of AI system activities to prepare for regular audits and inspections by regulatory authorities.

Do you remember, a few years ago, when all the talk was about us increasingly living in the virtual world? Where mixed reality living, powered by technology such as virtual reality (VR), was going to define how people lived, worked and played? So much so that fashion houses started selling in the virtual world. Estate agents started selling property in the virtual world and virtual conference centres were built so you could attend business events and network from the comfort of your office swivel chair. Futurists were predicting we were going to be living semi-Matrix-style in the near future.

Has it turned out like that? No… or certainly not yet anyway.

VR is just one example of how business is uniquely adept at propagating hype, particularly when it comes to emerging technologies. And you can probably guess where I am heading with this argument… AI.

The AI overhype cycle 

Since ChatGPT exploded into the public consciousness in 2022, I have spoken to scores of business leaders who feel like they need to jump on the AI bandwagon. It’s reflected by the last quarterly results announcements by the S&P 500, with over 40% of companies mentioning AI.  

They are understandably caught in the hype and buzz AI has created, and often think their businesses need to integrate this technology or face being left behind. This is reinforced by a recent BSI survey of over 900 leaders which found 76% believe they will be at a competitive disadvantage unless they invest in AI.

But is that true? The answer may be more nuanced than a simple yes or no.

To be clear, I am not saying the development of AI is anything but seismic. It is recognised by many leading academics as a general purpose technology (GTP). That is to say, it will be a game changer for humanity.

However, at an enterprise level, AI has been overhyped in many quarters, creating a disconnect between reality and expectations. 

Too much money for too little return 

This overhype is leading to two outcomes.

First, leaders feel pressured to be seen using it and heard talking about it. So they dabble with it, often without being certain how it will benefit their business, and how to effectively measure those benefits.

Second, the lack of a proper strategy and metrics is leading to time and resources being wasted. Just 44% of businesses globally have an AI strategy, according to the BSI survey. 

And importantly, if a user has a bad initial experience with a technology, it will often lead to mistrust and plummeting confidence in its future potential. This means it will take even more resources at a future date to effectively leverage the same technology. 

Recent media reporting has provided cases in point. There was the story of a chief marketing officer who abandoned one of Google’s AI tools because they disrupted the company’s advertising strategy so much, while another tool performed no better than a human. Then there was the tale of a chief information officer who dropped Microsoft’s Copilot tool after it created “middle school presentations”.

This disconnect is nothing new. As a consultant, what I often see is a detachment between a company’s business goals and how their technology is set up and operated. Or as in this case, a delta between expectations and delivery capability.

“Keep it simple” and focus on the business basics 

So amid all this noise around AI, my advice to clients is simple: keep in mind it is just another tool, and that the fundamentals of business haven’t changed.

You still need to provide a product or service that someone else wants to buy at a price point that is higher than what it costs to manufacture.

You still need to make a profit.

AI as a business tool may change the process by which we create and deliver value, but those business fundamentals haven’t changed and never will.

So if we recognise AI is just a tool, albeit one with the potential to accelerate the transformation of enterprises, what can leaders do to avoid landing in the gap between the hype and reality? Here are six suggestions:

1. Education

Invest in learning about the technology, its capabilities, the pros and cons, its roadmap and what dependencies AI has for it to be successful. Share this knowledge across the enterprise, so you start to take everyone on a collective journey

2. Build ethical AI policies and governance framework

Ethical AI policy is more than just guardrails to protect your business. It is also the north star that gives your employees, clients, partners, suppliers and investors confidence in what you will do with AI

3. Adopt a strategic approach

Focus on identifying key business problems where AI can be part of the solution. Put in place the appropriate metrics. This will help to prioritise investment and resource allocation

4. Develop your data strategy

AI success is intrinsically linked to data, so build your data strategy. Focus on building a solid data infrastructure and ensuring the quality of your data. This will lay the groundwork for successful AI implementation

5. Foster collaboration 

Consider collaborating with external partners, such as vendors or even universities and research institutions. This collective solving of problems will help provide deep insights into the latest AI developments and best practices

6. Communicate

Given the pace of business evolution nowadays, for most enterprises change management has become a core operational competency. So start your communication and change management early with AI. With its high public profile and fears persisting about AI replacing workers, you want to fill the knowledge gap in your team members so they understand how AI will be used to empower, not replace them. Taking employees on this journey will massively help the chances of success of future AI programmes.

Overall, unless leaders know how to integrate AI in a way that provides business benefits, they are just throwing mud at a wall and hoping some will stick… and all the while the cost base is rapidly increasing as a result of adopting this hugely expensive technology.

So to answer the big question, will a business be at a competitive disadvantage if it doesn’t invest in AI?

Typically, yes it will. But invest in a plan focused on how AI can help achieve longer-term business goals. Its capabilities will continue to emerge and evolve over the coming years, so building the right foundations will help effectively leverage AI both today and tomorrow.  

And ultimately remember that like all technology, AI is just one tool in the business kitbag.

Nigel O’Neill is founder and CEO of Tarralugo.

Since the launch of ChatGPT captured the global imagination, the technology has attreacted questions regarding its workings. Some of these questions stem from a growing interest in the field of AI design. Others are the result of suspicion as to whether AI models are being trained ethically.

Indeed, there’s good reason to have some level of skepticism towards generative AI. After all, current iterations of Large Language Models use underlying technology that’s extremely data-hungry. Even a cursory glance at the amount of information needed to train models like GPT-4 indicates that documents in the public domain were never going to be enough.

But I’m going to leave the ethical and legal questions for better-trained specialists in those specific fields and look at the technical side of AI. The development of generative AI is a fascinating occurence, as several distinct yet closely related disciplines had to progress to the point where such an achievement became possible.

While there are numerous different AI models, each accomplishing a separate goal, most of the current underlying technologies and requirements have many similarities. So, I’ll be focusing on Large Language Models as they’re likely the most familiar version of an AI model to most people.

How do LLMs work?

There are a few key concepts everyone should understand about AI models as I see many of them being conflated into one:

Large Language Model (LLM) is a broad term that describes any language model that uses a large amount of (usually) human-written text and is primarily used to understand and generate human-like language. Every LLM is part of the Natural Language Processing (NLP) field.

A Generative Pre-trained Transformer (GPT) is a type of LLM that was introduced by OpenAI. Unlike some other LLMs, the primary goal was to specifically generate human-like text (hence, “generative”). Pre-trained simply means that the model requires lots of labeled data to function.

Transformer is another part of GPT that people are often confused by. While GPTs were introduced by OpenAI, Transformers were initially developed by Google researchers in a breakthrough paper called “Attention is All You Need”.

One of the major breakthroughs was the implementation of self-attention. This allows a model that uses such a transformer to evaluate all words within it at once. Previous iterations of language models had numerous issues such as putting more emphasis on recent words.

While the underlying technology of a transformer is extremely complex, the basics are that they convert words (for language models) into mathematical vectors of three-dimensional space. Earlier iterations would only convert single words and place them in a three-dimensional space with some prediction if the words are related (such as “king” and “queen” being closer to each other than “cat” and “king”). A transformer is able to evaluate an entire sentence, allowing better contextual understanding.

Almost all current LLMs use transformers as their underlying technology. Some refer to non-OpenAI models as “GPT-like.” However, that may be a bit of an oversimplification. Nevertheless, it’s a handy umbrella term.

Scaling and data

Anyone who has spent some time analysing natural human language will quickly realize that language, as a concept or technology, is one of the most complicated things ever created. In fact, philosophers and linguists still spend decades trying to decipher even small aspects of natural language.

Computers have another problem – they don’t get to experience language as it is. So, like the aforementioned transformers, language has to be converted into a mathematical representation, which poses significant challenges by itself. Couple that with the enormous amount of complexities that our daily use of language has. From humor to ambiguity to domain-specific language – all of that adds to largely unspoken rules most of us understand intuitively.

Intuitive understanding, however, isn’t all that useful when you need to convert those rules into mathematical representations. So, instead of attempting to input rules to machines themselves, the idea was to give them enough data to glean out the intricacies of language. Unavoidably, however, that means that machine learning models have to acquire lots of different expressions, uses, applications, and other aspects of language. There’s simply no way to provide all of these within a single text or even a corpus of texts.

Finally, most machine learning models face scaling law problems. Most business-folk will be familiar with diminishing returns – at some point, each invested dollar into an aspect of business will start generating fewer returns. Machine learning models, GPTs included, face exactly the same issue. To get from 50% accuracy to 60% accuracy, you may need twice as much data and computing power than before. Getting from 90% to 95% may require hundreds of times more data and computing power than before.

Currently, the challenge seems largely unavoidable as it’s simply part of the technology, it can only be optimised.

Web scraping and AI

It should be clear by now that no matter how many books were written before the invention of copyright, there wouldn’t nearly be enough data for models like GPT-4 to exist. The enormous requirements of data, and the existence of an OpenAI web crawler, outside of publicly available datasets, OpenAI (and likely many of their competitors) likely used web scraping to gather the information they needed to build their LLMs.

Web scraping is the process of creating automated scripts that visit websites, download the HTML file, and store it internally. HTML files are intended for browser rendering, not data analysis, so the downloaded information is largely gibberish. Web scraping systems also have a parsing aspect that fixes the HTML file so that only the valuable data remains. Many companies use already use these tools to extract information such as product pricing or descriptions. LLM companies parse and format content in such a way that it resembles regular text like a blog post. Once a website has been parsed, it’s ready to be fed into the LLM.

All of this is used to acquire the contents of blog posts, articles, and other textual content. It’s being done at a remarkable scale.

Problems with web scraping

However, web scraping runs into two issues. One, websites aren’t usually all that happy about a legion of bots sending thousands of requests per second. Second, there is the question of copyright. Most web scraping companies use proxies, intermediary servers, that make changing IP addresses easy, which circumvents blocks, intentional or not. Additionally, it allows companies to acquire localised data – extremely important to some business models such as travel fare aggregation.

Copyright is a burning question in both the data acquisition and AI model industry. While the current stance is that publicly available data, in most cases, is alright to scrape, there’s questions about basing an entire business model that, in some sense, uses the data to replicate the text through an AI model.

Conclusion

There are a few key technologies that have collided to create the current iteration of AI models. Most of the familiar ones are based on machine learning, particularly the transformer invention.

Transformers can take textual data and convert it into vectors, however, their key advantage is the ability to take larger pieces of text (such as sentences) and look at them in their entirety. Previous technologies usually were only capable of evaluating words themselves.

Machine learning, however, has the problem of being data-hungry and exponentially-so. Web scraping was utilized in many cases to acquire terabytes of information from publicly available sources.

All of that data, in OpenAI’s case, was cleaned up and fed into a GPT. They are then often fine-tuned through human intervention to get better results out of the same corpus of data.

Inventions like ChatGPT (or chatbots with LLMs in general) are simply wrappers that make interacting with GPTs a lot easier. In fact, the chatbot part of the model might just be the simplest part of it.

In an era which demands flexibility and fast-paced responses to cyber threats and sudden market shifts, risk management has never been in more need for tools to support its ever-evolving transformation. 

AI is the key player which can keep up and perform beyond expectations. 

This isn’t about flashy tech for tech’s sake; rather, it’s about harnessing tools that can make businesses more resilient and agile. Sounds complicated? It’s not.  Here’s how your company can manage risk with ease and let your business grow with AI. 

Why should I care?

Put simply, AI-driven automation involves using technology to perform tasks that were traditionally done by humans, but with added intelligence. 

Unlike basic automation that follows set instructions, AI systems learn from data, recognise patterns, and even make decisions. In risk management, this means AI can help identify potential risks, assess their impact, and even respond in real time—often faster and more accurately than human teams.

Think of it like this: In finance, AI can monitor market fluctuations and automatically adjust portfolios to reduce exposure to risk. In operations, it can predict supply chain disruptions and recommend alternative strategies to keep production on track. AI helps by doing the heavy lifting, leaving leaders with clearer insights and the ability to make more informed decisions.

The insurance industry is a stand-out example of how AI-powered risk management can be done. It is transforming the sector by streamlining underwriting and claims processing, making confusing paperwork a thing of the past and loyal customers a thing of the future.

The Potential

Risk is part of doing business. We all know that, but the nature of risk has evolved, calling into question just how much companies can tolerate. Thanks to the interconnectedness of our digital and global economies, we can make fewer compromises and implement effective coping strategies to mitigate potential disruption which can ripple within minutes. 

For example, if you are a large international organisation, AI-driven automation can prove to be a valuable assistant when dealing with regulatory changes. JP Morgan jumped at the chance to incorporate AI’s uses. It has integrated AI into its risk management processes for fraud detection and credit risk analysis. The bank uses machine learning algorithms to analyse vast amounts of transaction data, detecting unusual patterns and flagging potentially fraudulent activities in real time. This has helped them significantly reduce fraud losses and improve the efficiency of their internal audit processes.

Additionally, the pace at which data is generated has exploded, making it nearly impossible for traditional risk management processes to keep up. 

This is where AI’s ability to process vast amounts of data quickly and accurately comes in handy. It offers predictive power that helps leaders anticipate risks instead of reacting to them. AI doesn’t get overwhelmed by the volume of information or distracted by the noise of the day; it consistently analyses data to identify potential threats and opportunities.

The automation aspect ensures that once risks are identified, responses can be triggered automatically. This reduces the chance of human error, speeds up reaction times, and allows teams to focus on strategic tasks rather than manual monitoring and troubleshooting.

The limitations

While a powerful tool, it doesn’t make it invincible or infallible. 

To ensure proper implementation, leaders must take note of its limitations. This means rolling out training across company departments to educate and upskill staff. This can involve conducting workshops, recruiting AI experts to the team, and setting realistic expectations from day one about what AI can and can’t do.

By teaming up with AI, company leaders can create a sandbox environment where you interact with AI using your own data. This practical approach simplifies the transition more than a lecture in a seminar room and can be tried and tested without full commitment or investment.

How AI Automation Can Make an Impact

There are several critical areas where AI-driven automation is already making a significant impact in risk management:

Cybersecurity is a sector that has huge potential for growth. As cyber threats become more sophisticated, AI systems are helping companies defend themselves. These systems can identify patterns of malicious behaviour, recognise the latest attack methods, and automate responses to neutralise threats quickly. 

This reduces downtime and limits damage, allowing companies to stay one step ahead of hackers. AXA has developed AI-powered tools to manage and mitigate cyber risks for both its operations and its customers. By leveraging AI, AXA analyses vast amounts of network data to detect and predict cyber threats. This helps businesses proactively manage vulnerabilities and minimise cyberattacks. 

The regulatory landscape is constantly shifting, and keeping up with these changes can be overwhelming. AI can automate the process of monitoring new regulations, assess their impact on the business, and ensure compliance by flagging potential issues before they become problems. This is especially critical for industries like finance and healthcare, where non-compliance can result in heavy fines or legal trouble.

Supply Chain Management also benefits from its implementation. Walmart uses AI to monitor risks in its vast network of suppliers. The company has developed machine learning models that analyse data from its suppliers, including financial stability, production capabilities, and past performance. AI also evaluates external data sources such as economic indicators, political risks, and natural disasters to identify potential threats to supply chain continuity.

How Leaders Can Implement AI-Driven Automation in Risk Management

How to embrace its innovation:

Identify Key Risk Areas: Start by mapping out the areas of your business most susceptible to risk. Whether it’s cybersecurity, regulatory compliance, financial instability, or operational inefficiencies, knowing where the biggest vulnerabilities lie will help you focus your AI efforts.

Assess Current Capabilities: Look at your current risk management processes and assess where automation could provide the most value. Are your teams spending too much time monitoring data? Are there manual tasks that could be streamlined? AI can enhance these processes by improving speed and accuracy.

Choose the Right Tools: Not all AI solutions are created equal, and it’s essential to choose tools that fit your specific needs. Work with trusted vendors who understand your industry and can offer customised solutions. Look for AI systems that are transparent, explainable, and adaptable to evolving risks.

Monitor and Adapt: AI systems need regular updates and monitoring to remain effective. Make sure you have a plan in place to review performance, adjust algorithms, and update data sets. This will ensure your AI tools continue to provide relevant, actionable insights as risks evolve.

If you don’t have the right talent, or capacity, or you’re unsure where to start, choose a reliable partner to help accelerate your use case and really get the best out of it. 

AI-driven automation is reshaping the future of risk management by making it more proactive, predictive, and efficient. Company leaders who embrace these technologies will not only be better equipped to navigate today’s complex risk landscape but will also position their businesses for long-term success. 

According to Forbes Advisor, 56% of businesses are using AI to improve and perfect business operations. Don’t risk falling behind and discover the wonders of AI today.

In this day and age, it’s safe to say we’re drowning in data. Every second, staggering amounts of information are generated across the globe—from social media posts and news articles to market transactions and sensor readings. This deluge of data presents both a challenge and an opportunity for businesses and organisations. The question is: how can we effectively harness this wealth of information to drive better decision-making?

As the founder of Permutable AI, I’ve been at the forefront of developing solutions to this very problem. It all started with a simple observation: traditional data analysis methods were buckling under the sheer volume, velocity, and variety of modern data streams. The truth is, a new approach was needed—one that could not only process vast amounts of information but also extract meaningful insights in real-time.

Enter AI 

Artificial Intelligence, particularly ML and NLP, has emerged as the key to unlocking the potential of big data. At Permutable AI, we’ve witnessed firsthand how AI can transform data overload from a burden into a strategic asset.

Consider the financial sector, where we’ve focused much of our efforts. There was a time when traders and analysts would spend hours poring over news reports, economic indicators, and market data to make informed decisions. In stark contrast, our AI-powered tools can now process millions of data points in seconds, identifying patterns and correlations that would be impossible for human analysts to spot.

But this isn’t just because of speed. The real power of AI lies in its ability to understand context and nuance. And this isn’t just about systems that can count keywords; they can also comprehend the sentiment behind news articles, social media chatter, and financial reports. This nuanced understanding allows for a more holistic view of market dynamics, leading to more accurate predictions and better-informed strategies.

AI’s Impact across industries

Needless to say, this transformation isn’t just limited to the financial sector, because the reality is AI is transforming how data is gathered, processed and used  across various sectors. Think of the potential for AI algorithms in analysing patient data, research papers, and clinical trials to assist in diagnosis and treatment planning. 

During the COVID-19 pandemic, while we were all happily – or perhaps not so happily, cooped up indoors, we saw how AI could be used to predict outbreak hotspots and optimise resource allocation. Meanwhile, the retail sector is already benefiting from AI’s ability to analyse customer behaviour, purchase history, and market trends, providing personalised product recommendations that are far too tempting, as well as optimising inventory management.

The list goes on, but in every sector, and in every use case, there is the potential here to not replace human expertise, but augment it. The goal should be to empower decision-makers with timely, accurate, and actionable insights, because in my personal opinion, a safe pair of human hands is needed to truly get the best out of these kinds of deep insights. 

Overcoming challenges in AI implementation

Despite its potential, implementing AI for data analysis is not without challenges. In my experience, three key hurdles often arise. Firstly, data quality is crucial, as AI models are only as good as the data they’re trained on. Ensuring data accuracy, consistency, and relevance is paramount. Secondly, as AI models become more complex, explaining their decisions becomes more challenging. 

This means investing heavily in developing explainable AI techniques to maintain transparency and build trust – and the importance of this can not be understated. AI plays an increasingly significant role in decision-making, addressing issues of bias, privacy, and accountability will become ever more crucial. With that said, overcoming these challenges requires a multidisciplinary approach, combining expertise in data science, domain knowledge, and ethical considerations.

The Future of AI-Driven Data Analysis

Looking ahead, I see several exciting developments on the horizon. Federated learning is a technique that allows AI models to be trained across multiple decentralised datasets without compromising data privacy. 

It could unlock new possibilities for collaboration and insight generation. Then, as quantum computers become more accessible, they could dramatically accelerate certain types of data analysis and AI model training. Automated machine learning tools will almost certainly democratise AI, allowing smaller organisations to benefit from advanced data analysis techniques rather than it just being the playground of the big boys.

 Finally, Edge AI, which processes data closer to its source, will enable faster, more efficient analysis, particularly crucial for IoT applications.

Navigating the AI future 

One thing if for certain, the data deluge shows no signs of slowing down. But with AI, what once seemed like an insurmountable challenge is now an unprecedented opportunity. By harnessing the power of AI, organisations can turn data overload into a wellspring of strategic insights.

It’s important to remember that the future of business intelligence is not just about having more data; it’s about having the right tools to make that data meaningful. In this data-rich world, those who can effectively harness AI to cut through the noise and extract valuable insights will have a decisive advantage. The question is no longer whether to embrace AI-driven data analysis, but how quickly and effectively we can implement it to drive our organisations forward.

To be clear, the competition is fierce in this rapidly evolving field. But while challenges remain, the potential rewards are immense. The reality is that AI-driven data analysis is becoming increasingly important across all sectors. For now, we’re just scratching the surface of what’s possible. As so often happens with transformative technologies, we’re likely to see even more remarkable insights emerge as AI continues to evolve. But it’s important to remember that AI is a tool, not a magic solution. 

Embracing the AI-driven future

As it stands, nearly every industry is grappling with how to make the most of their data. As for the future, it’s hard to predict exactly where we’ll be in five or ten years. Today, we’re seeing AI make a big splash in fields from finance to healthcare. The concern for people often centres around job displacement. However, all this means is that we need to focus on upskilling and retraining to work alongside AI systems.

And that’s before we address the potential of AI in tackling global challenges like climate change or pandemics. It’s the same story on a smaller scale in businesses around the world. AI is helping to solve problems and create opportunities like never before.

Ultimately, we must remember that the goal of all this technology is to enhance human decision-making, not replace it. It’s no secret that the world is becoming more complex and interconnected. In large part, our ability to navigate this complexity will depend on how well we can harness the power of AI to make sense of the vast amounts of data at our fingertips.

At the end of the day, AI-driven data analysis is not just about technology—it’s about unlocking human potential. And that, to me, is the most exciting prospect of all.

Data analytics has truly gone mainstream. Organisations across the world, in nearly every industry, are embracing the practice. Despite this, however, the execution of data analytics remains varied – and not all data analytics approaches are made equal.

For most organisations, the most advanced data analytics team is  the centralised Business Intelligence (BI) team. This isn’t necessarily inferior to having a specialist data science team in place. However, the world’s most successful BI teams do embrace data science principles. Comparatively, this isn’t something that all ‘classic BI teams’ nail. 

With more and more mature organisations benefiting from best practice data analytics – competitors that haven’t adapted risk getting left in the dust. The charter and organisation of typical BI need to be set up correctly for data analytics to address increasingly complicated challenges and drive transformational change across the business in a holistic manner.

Where is classic BI lacking?

BI’s primary focus is descriptive analytics. This means summarising what has happened and providing visualisation of data through dashboards and reports to establish trends and patterns. Visualisation is foundational in data analytics. The problem lies in how this visualisation is being carried out by BI teams. It’s often the case that BI teams are following an IT project model. They churn out specific reports like a factory production line based on requirements set by another part of the business. Too often, the goal is to deliver outputs quickly in a visually appealing way. However, this approach has several key deficiencies.

Firstly, it’s reactive rather than proactive. It is rooted in delivering reports or visualisations that answer predefined questions framed by the business. This is opposed to exploring data to uncover new insights or solve open-ended problems. This limits the potential of analytics to drive new innovative solutions.

Secondly, when BI teams follow an IT project model, they typically report to central IT teams rather than business leads. They lack the authority to influence broader business strategy or transformation. Therefore, their work remains siloed and disconnected from the core strategic objectives of the organisation. For too many companies, BI has remained a tool for looking backwards, rather than a driver of forward-thinking, data-driven decision-making. The IT model of collecting requirements and building to specification is not the transformational process used by world-class data science teams. Instead, understanding the business and driving change is a central theme seen within the world’s leading analytic organisations. 

The case for centralisation

To unlock the full potential of data analytics, organisations must centralise their data functions. They need a simple chain of command that feeds directly into the C-Suite. Doing so aligns data science with the business’s strategic direction. Doing so successfully creates several advantages that set companies with world-class data analytics practices apart from their peers.

Solving multi-domain problems with analytics

A compelling argument for centralising data science is the cross-functional nature of many analytical challenges. For example, an organisation might be trying to understand why its product is experiencing quality issues. The solution might involve exploring climatic conditions causing product failure, identifying plant processes or considering customer demographic data. These are not isolated problems confined to a single department. The solution therefore spans multiple domains, from manufacturing to product development to customer service.

A centralised data science function is ideally positioned to tackle such complex problems. It can draw insights from various domains as an integrated team to create holistic solutions without different parts of the organisation working at odds with each other. In contrast, where data scientists report to individual departments (centralisation isn’t happening) there’s a big risk of duplicating efforts and developing siloed solutions that miss the bigger picture.

Creating career pathways and developing talent

It should be obvious to state – data scientists need career paths too. The most important asset of any data science domain is the people. But despite this, where teams are decentralised, data scientists tend to work in small, isolated teams within specific departments. This limits their exposure to a broader range of problems and stifling career advancement opportunities. 

For example, a data scientist in a three-person marketing analytics team has fewer opportunities and less interaction with the overall business than a member of a 50-person corporate data science team reporting to the C-suite.

Centralising the data science team within a single organisational structure enables a more robust career path and fosters a culture of continuous learning and professional development. 

Data scientists can collaborate across domains, learn from each other and build a diverse skill set that enhances their ability to tackle complex problems. Moreover, it’s easier to provide consistent training, mentorship and development opportunities where data science is centralised, ensuring that teams are fully equipped with the latest tools and techniques.

Linking analytics across the business

A centralised data science function acts as a valuable bridge across different parts of the business. Let’s take an example. Two departments approach the data science team with seemingly conflicting requests. 

The supply chain team wants to minimise shipment costs and asks for an analytic that will identify opportunities to find new suppliers near existing manufacturing facilities. 

The purchasing team, separately, approaches the data science team to reduce the cost of each part. To do this, they want to identify where they have multiple suppliers, and move to a model with a single global supplier that has much larger volumes and will reduce costs. These competing philosophies will each optimise a piece of the business, but in reality, what should happen is a single optimised approach for the business.

Instead of developing competing solutions, a centralised data science team can balance competing objectives and deliver an optimal solution that’s aligned with overall strategy. Cast in this role, data science is the strategic partner contributing to the delivery of the best outcomes for the organisation.

Leveraging analytics methods across domains

The best breakthroughs in analytics come not from new algorithms, but from applying existing methods to innovate use cases. 

A centralised data science team, with its broad view of the organisation’s challenges, is more likely to recognise these opportunities and adapt solutions from one domain to another. For example, an algorithm that proves successful in optimising marketing campaigns could be adapted to improve inventory management or streamline production processes.

Driving organisational change and analytics maturity

Finally, a centralised data science function is best positioned to drive the overall analytic maturity of the organisation. 

This function can standardise governance, as well as best practices. In doing so, it can drive the change management processes, ensuring that data-driven decision-making becomes ingrained in company culture. 

The way forward

The shift from classic BI to a centralised data science function is not just a structural change; it is a crucial strategy for companies looking to stay ahead in a competitive, data-driven landscape. By centralising data science and enforcing a charter for BI to solve key problems of the organisation rather than be dictated to, companies can solve complex, cross-functional problems more effectively, foster talent development, create inter-departmental synergies and drive a culture of continuous improvement and innovation. 

This evolution is what sets world-class companies apart from the rest. It might just be the transformation your company needs to unlock its full potential.