The growing adoption of artificial intelligence (AI), combined with advances in the technology and the recent denial of patents to an AI inventor by the European and American patent offices, are necessitating a critical examination of the profound role the IP system has on AI, not only because IP rights can protect AI-related inventions and creations, but also because they can block access to key AI technologies. But such a review raises complicated IP, technical, legal, data-related, social and philosophical issues. For example, if the IP system is designed to encourage creation and invention, in large part through the recognition and reward of creative and inventive endeavour, who or what should be allowed to benefit from it – a human with an artificial identity; a human enhanced with AI; or an AI itself? And if not an AI, then what types of inventions or creations, created with the assistance of AI, deserve IP protection? This chapter attempts to help readers better appreciate the emerging and complex questions regarding IP and AI including what IP is in the context of AI, why things have gotten so complicated and the conundrum we now face of whether to prioritise progress or human interests.

This chapter calls for caution in the US drive to negotiate new trade provisions to govern the digital economy. It is in four parts. Part I introduces the transformative nature of innovation in the data-driven economy. Part II presents the severe challenges posed by this development, including its contribution to monopolization, inequality, social control through surveillance, geopolitical competition between the United States and China, and system vulnerability. Part III assesses the current trade negotiating context involving competing models advanced by the United States (advocating free data flow), the European Union (promoting privacy regulation), and China (stressing sovereignty and the facilitation of trade in goods through e-commerce). Part IV presents a governance framework that calls for modesty in recognition of national differences and the severe challenges posed, while building institutions to foster deliberation and learning in light of uncertainty.

What is a taxonomy?

According to Collins Dictionary (2021), the definition of a taxonomy is ‘the process of naming and classifying things such as animals and plants into groups within a larger system, according to their similarities and differences’.

That's a very broad and vague description and I’d say a taxonomy is more than that. In my spend classification world, a taxonomy is a list of categories and sub-categories used to classify data into groups or ‘buckets’ of items or services, providing a unified view for the purpose of reporting and analytics. It can be used by the whole business or just one department, such as procurement. In procurement, a taxonomy can also be known as a ‘category’ or ‘spend tree’.

You might not realise it, but taxonomies are used in many everyday activities, for example online shopping. Every time you do your online food shop, you are inadvertently using a taxonomy. If you need to search for apples, you might go to Fresh Food, followed by Fruit, followed by Apples. That's your level 1, 2 and 3. Try looking on another supermarket's website and you’ll find the apples in a different category structure.

This is reflective of spend taxonomies in business too. You’ll find many different variations, all being used with differing perspectives. If you move from company A to company B, you might find that everything you knew is now different in your new company.

A taxonomy usually has several levels and the number can range from 1–6 (although I have heard of more, can you believe?!). I would recommend 3–4 levels as the optimal option, the reason being that most organisations do not interrogate their data to any further level of detail. Also, the more detailed the taxonomy, the longer the classification process takes and the more open it is to mistakes.

Think of peripherals: if you can classify mouse, keyboard, monitor and docking station all as peripherals, you can classify that pretty quickly. However, if you have to search for each item individually, and classify it individually, it adds time to the process.

Hello and welcome to Between the Spreadsheets: Classifying and Fixing Dirty Data.

This is not your typical data book and that's because I’m not your typical data person. I have a wonderfully unique background that's a mix of corporate and data work, which has brought me to the point where I’m able to share my specialist knowledge with you.

Regardless of whether you are completely intimidated by data, starting out in your data career, a seasoned procurement or data professional, or a decision maker within an organisation, there will be something in here for you.

Dirty data is a problem. In every single organisation, no matter how big or small or where they’re located, you will hear talk of data quality issues. What you will rarely hear about is the consequences of this because people or companies don't want to admit their failures. We could be talking about millions of pounds or dollars lost on new technology, weeks or months spent fixing mistakes due to bad data, possible job losses or even worse.

It's not just that. We hear all the time of data scientists spending anything from 40–80% of their time cleaning or wrangling data. Why is this? Well, I believe it's because they are inefficient and inexperienced at it. ‘What? But they’re data scientists!’ I hear you cry. Unfortunately, that doesn't mean anything. Data cleaning is rarely covered in academic studies or other courses; the focus is always on the technical aspects of the role, yet ironically, they can't do any of that without the clean data first.

While data cleaning is one of the most vital parts of the whole process when working with data, it is often overlooked because either there is an assumption that people already know how to do it or it's considered too menial or not important enough to spend time on, or invest in.

It's not just in data science; I’ve seen this in other areas. I was managing a politics student and asked her to book a meeting and send invites. When it hadn't been done, I asked why and she explained she didn't know how to do it. I was blown away that someone studying politics (don't ask me why the politics thing made a difference, I have no idea) didn't know how to create a meeting invite. My assumptions were very wrong.

What is dirty data?

Let's start on some common ground. What exactly is dirty data?

Well, the truth is that it can mean different things to different people working with different types of data. For the purposes of this book, it will be based on information used in a business context.

At its most basic level, dirty data is anything incorrect. It could be things such as:

Misspelt names

This happens more than you think. If it's supplier names, it could be a simple switch of letters from ABC Printing to ABC Printign, a missing letter such as T Shoesmit instead of T Shoesmith, or something much more subtle like AT Jones, instead of TA Jones, which may not be easily picked up.

If you’re dealing with personal information, it's doubly important to get the name right because of data protection regulations, such as the General Data Protection Regulation (GDPR). Very recently, I received a piece of mail for my new limited company ‘The Classification Guru Ltd’. The address was correct and my first and middle names were correct, but I had someone else's surname and a business name that wasn't mine.

When I checked on Companies House, I could see that the surname and the business name were related to one person – everything else was my information. What I suspect happened in this instance is one of several things: firstly that the list of names for mailings was in Excel and someone possibly hadn't filtered all columns and the information was therefore mixed up. Secondly, it could have been that some lines of data were removed, which caused some of the information in certain columns to shift up or down and misalign. It could have been something as simple as a cut and paste error that caused the problem. This could have easily been rectified by applying some spot checks to the data before it was used as a mailing list. I’ll cover this further later.

Incorrect or misleading descriptions

In the work I do, I see this a lot in invoice or Purchase Order (PO) descriptions. It could be something as simple as ‘services’ in the description and the person's name as the supplier.

Cleansing personal data

We have focused a lot on classification and normalisation, but what about other data cleansing such as supplier or customer addresses? I have seen first hand how many duplicates can occur if you’re working with multiple systems.

I had a client with nine different sources of data, which when merged totalled 2.8 million rows of customer information. When I cleansed this, it reduced to 1.3 million rows of data. I certainly didn't do this in Excel, and there are a number of tools available that can help, but if you’re on a budget or need to get something done relatively quickly, I can show you how to achieve this using Excel.

It would be hard to cover all the different formats that addresses can come in. Quite often you will see a mixture of formats, such as everything in one cell or spread across many cells or with the information in the wrong column. I’ll be focusing on names, addresses and e-mails for this exercise, but you will most likely have telephone numbers as well.

I’ll share as an example what I’m going to call my sample customer list (Figure 5.1 on the next page). It's got a name and address and an e-mail column and there is a mixture of upper and lower case data, as well as sentence case. This is not unusual and I see this in a lot of files. Again, it's down to creating some standards and making sure everyone applies them.

As all the formulas used for this process are in previous chapters, there will be less illustrations in this chapter, but feel free to refer back to Chapters 1, 2 and 4 if you’d like some visual support.

Cleansing names in Excel

There are a number of different ways to approach this and it very much depends on what you need for your business. In my sample file, the names are all in one column. Is that how you want them to stay? Or do you want to split them out into first name and last name? This is most likely something that is decided elsewhere in the business, but it doesn't mean you can't use their standards and formatting and be consistent.

We’ve covered a lot in this book, so it is helpful to summarise and pull together the main points.

I hope you’ve enjoyed reading this book and have been able to take something away from it to improve your data quality. I’ve certainly enjoyed writing it – a lot more than I thought I would! – and I have really surprised myself with how much I can actually write about data. There's another tip for you: even if you think you can't – try! You’ll be amazed at what you’re capable of.

The same goes with data normalisation, classification and cleansing. You might think it's a mammoth task and that it's too big to deal with, but you can start small by focusing on the high value spend or customers or the most frequently used data and work from there.

Dirty data

I’ve shared with you the types of dirty data you might find, and of course, there will be some specific examples related to your organisation. But in general, it will be things like: misspelt names, incorrect or misleading descriptions, missing or incorrect codes, no standard formatting of addresses or units of measure, currency issues, incorrect or partially classified spend data, and not forgetting the most popular, the duplicates.

In terms of the consequences of dirty data, I’ve given examples of what could go wrong and what has gone wrong for some of my clients in the past. Fear not though, you now have spot-checking tips for your data toolbelt.

COAT

What goes better with your data toolbelt than your data COAT? Your data needs to be consistent, organised, accurate and trustworthy and you have to have all four as they are all interdependent. Make COAT fun so that you can engage non-data people within your organisation in a meaningful way and relate it to any data situation.

Normalisation

When you’re ready to add to your data toolbelt, you can start to normalise your suppliers. The key message here is ‘rinse and repeat’ – don't just normalise once and think that's it. As you are cleaning up the data, you can create new near-duplicates that need to be tidied up, so go back and check a second or third time.

I’ve shared a lot of knowledge with you in this book on how to improve your spend data, but how do you know where you sit now and what needs to be improved? In this chapter, I’ll introduce you to a maturity model that will help you answer those questions. Not one of those fancy consultancy models that has big words and is very technical, but one that is relatable, easy to understand and that you could give to anyone in your organisation and they would have some basic understanding of it. The inspiration for the model came from COAT.

Let me introduce you to …

The dirty data maturity model

Let me share with you how you can move from dirty data to dirt-free data using this model, charting where you are, where you’d like to be and how to get there. We all want dirt-free data, but that could take years to achieve, so let's be realistic about what you have and what you could have in the next 12–24 months.

Dirty data

Dirty data is a pretty bad situation. It's the equivalent of turning your underwear inside out to wear it for another day – you know you have to do something about it, but you can get away with it for a little bit longer. You are delaying the inevitable. What if that one extra day you wear those underpants, you have an accident and get taken to hospital? That's when you get found out.

Data is no different. It is widely known, and I hate to say accepted, within organisations that there are data issues internally, yet no one really wants to address them. Why? Generally, it's not an easy fix, the business might not see data as an investment and the problem could be so big that there's just not the resources internally to fix it and no one wants to pay for a third party to do it. Then the data accident happens and everything gets exposed. At this point, there's no hiding from it. Fingers will be pointed, blame will be apportioned and everyone will be thinking ‘why didn't we just fix this earlier?’

What is supplier normalisation?

Let's start with why I’m referring to ‘supplier normalisation’ and not ‘vendor normalisation’. I work with companies all over the world and I see the use of ‘supplier’ a lot in the UK and Europe, but ‘vendor’ occurs more frequently in the US. In my day to day, I alternate between supplier and vendor and use them interchangeably.

I created a poll on LinkedIn and asked my network what their preferred terminology is. I had no idea that it would cause such a debate! 1,184 people voted, over 200 commented and the final result was: 65% voted ‘supplier’ and 35% voted ‘vendor’. I had some great comments for justification of each, including:

As you can see, it's not clear cut and both terms are widely accepted. For the purpose of this book, I will use supplier, which will also encompass anything vendor related.

Supplier normalisation can also be referred to as ‘standardisation’. Normalisation is giving your existing suppliers a common name or grouping so that you can see how much you are truly spending with them or selling to them. When you are doing this for the first time, I would always recommend adding a new column to your spreadsheet. I would not recommend overwriting your existing suppliers as you may need this information in the future, and it will definitely help for classification, which I’ll cover in Chapter 4.