The IDI turns 10: Why the world’s watching New Zealand
NZ’s world-leading data system is transforming social investment, helping policymakers and organisations make smarter, evidence-based decisions.
On August 16th, Stats NZ’s Integrated Data Infrastructure (IDI) turns 11.
The IDI is the largest research database in the country. It combines 29 databases from different government departments, containing eight billion data points on nine million Kiwis stretching back to the 1980s.
Before the IDI, it was extremely difficult for researchers and policymakers to understand New Zealanders’ interactions with different government services and what these interactions might suggest about their life outcomes.
Researchers may have hypothesised that giving someone social housing would decrease the likelihood of them spending time in jail, but it was difficult to prove this link because they couldn’t see data about the same individual from Kāinga Ora and the Ministry of Corrections.
The IDI changed this by linking information about people’s education, health, income, and more.
As an example, the Social Investment Agency showed that people who received a social house spent less time in jail compared to a trial group with similar characteristics using IDI data which linked people’s interactions with Kāinga Ora and Ministry of Corrections.
It’s not just the Social Investment Agency - any approved researcher or policymaker can access the linked data in the IDI to help them make more evidence-based, informed decisions.
Combining the Best of Both Worlds
Let’s address the elephant in the room: privacy. Thankfully, the IDI does a great job at protecting people’s privacy while producing valuable insights.
In New Zealand, individuals have different identifiers for each government service. The health system uses an NHI, education uses NSN, Inland Revenue has unique IRD numbers, and so forth. This approach has big privacy advantages compared to single identifier systems like the United States social security number. If you have your social security number stolen, as happens to nine million Americans each year, it can seriously affect your ability to access government services.
Stats NZ takes individuals' identifiers from different government departments and links them together in the IDI. It then removes all the identifying information to maintain the confidentiality of the individual.
This process achieves the best of both worlds. It helps us understand New Zealanders better so we can make better policy decisions, without needing to sacrifice our privacy by using a single identifier.
While the average New Zealander may not understand how special the IDI is, other governments do. Stats NZ regularly meets with other countries attempting to construct their own IDI equivalents. These countries have also started catching up in recent years - the Administrative Data Research in the UK being one such example.
As the IDI enters its second decade, there are a number of improvements that would help it continue to stand out globally. One of the main limitations we’ve seen at ImpactLab is with disability data.
The IDI only records whether a person is disabled or not, rather than information about the type and extent of their disability. While acknowledging that classifying disabilities is a tricky and sensitive area, we believe the status quo disadvantages some government and community disability programmes. A lack of information about the life outcomes of people with different kinds of disabilities, makes it more difficult to understand the potential long-term social impact of programmes that work with significantly disabled people.
In the world of statistics, as with the Olympics, New Zealand punches well above its weight.
The IDI, with its careful balance between actionable data and privacy, is the gold standard.
Call us naive - or just plain nerdy - but we believe it’ll go down as one of our greatest exports to the world in the last 30 years.
Here’s to many more birthdays!
For the policymakers:
The IDI is the secret sauce that makes social investment possible and contains evidence that can be used to make better policy decisions.
Example:
The Social Investment Agency showed how social housing is associated with a reduction in the amount of time people spend in jail.
For the analysts:
The IDI can be used by any approved individual. It can help you better understand Kiwis’ interactions with different government services and what these interactions might suggest about their life outcomes. While interactions in the IDI often just show the tip of the iceberg, they can point towards important realities in peoples’ lives.
Example:
A Treasury analysis using IDI found that for children 14 and under, there are four key indicators associated with poor life outcomes:
- Having a CYF finding of abuse or neglect
- Being mostly supported by benefits since birth
- Having a parent with a prison or community sentence
- Having a mother with no formal qualifications
The more of these indicators that a child experiences, the higher the association with poor future outcomes.
For the frontline workers:
We’ve heard from many frontline workers that when you work with people, it’s important to consider what’s going on across multiple aspects of their lives.
Tools like the IDI can help us understand more holistically the situations people face, and the difference you can make.
On August 16th, Stats NZ’s Integrated Data Infrastructure (IDI) turns 11.
The IDI is the largest research database in the country. It combines 29 databases from different government departments, containing eight billion data points on nine million Kiwis stretching back to the 1980s.
Before the IDI, it was extremely difficult for researchers and policymakers to understand New Zealanders’ interactions with different government services and what these interactions might suggest about their life outcomes.
Researchers may have hypothesised that giving someone social housing would decrease the likelihood of them spending time in jail, but it was difficult to prove this link because they couldn’t see data about the same individual from Kāinga Ora and the Ministry of Corrections.
The IDI changed this by linking information about people’s education, health, income, and more.
As an example, the Social Investment Agency showed that people who received a social house spent less time in jail compared to a trial group with similar characteristics using IDI data which linked people’s interactions with Kāinga Ora and Ministry of Corrections.
It’s not just the Social Investment Agency - any approved researcher or policymaker can access the linked data in the IDI to help them make more evidence-based, informed decisions.
Combining the Best of Both Worlds
Let’s address the elephant in the room: privacy. Thankfully, the IDI does a great job at protecting people’s privacy while producing valuable insights.
In New Zealand, individuals have different identifiers for each government service. The health system uses an NHI, education uses NSN, Inland Revenue has unique IRD numbers, and so forth. This approach has big privacy advantages compared to single identifier systems like the United States social security number. If you have your social security number stolen, as happens to nine million Americans each year, it can seriously affect your ability to access government services.
Stats NZ takes individuals' identifiers from different government departments and links them together in the IDI. It then removes all the identifying information to maintain the confidentiality of the individual.
This process achieves the best of both worlds. It helps us understand New Zealanders better so we can make better policy decisions, without needing to sacrifice our privacy by using a single identifier.
While the average New Zealander may not understand how special the IDI is, other governments do. Stats NZ regularly meets with other countries attempting to construct their own IDI equivalents. These countries have also started catching up in recent years - the Administrative Data Research in the UK being one such example.
As the IDI enters its second decade, there are a number of improvements that would help it continue to stand out globally. One of the main limitations we’ve seen at ImpactLab is with disability data.
The IDI only records whether a person is disabled or not, rather than information about the type and extent of their disability. While acknowledging that classifying disabilities is a tricky and sensitive area, we believe the status quo disadvantages some government and community disability programmes. A lack of information about the life outcomes of people with different kinds of disabilities, makes it more difficult to understand the potential long-term social impact of programmes that work with significantly disabled people.
In the world of statistics, as with the Olympics, New Zealand punches well above its weight.
The IDI, with its careful balance between actionable data and privacy, is the gold standard.
Call us naive - or just plain nerdy - but we believe it’ll go down as one of our greatest exports to the world in the last 30 years.
Here’s to many more birthdays!
For the policymakers:
The IDI is the secret sauce that makes social investment possible and contains evidence that can be used to make better policy decisions.
Example:
The Social Investment Agency showed how social housing is associated with a reduction in the amount of time people spend in jail.
For the analysts:
The IDI can be used by any approved individual. It can help you better understand Kiwis’ interactions with different government services and what these interactions might suggest about their life outcomes. While interactions in the IDI often just show the tip of the iceberg, they can point towards important realities in peoples’ lives.
Example:
A Treasury analysis using IDI found that for children 14 and under, there are four key indicators associated with poor life outcomes:
- Having a CYF finding of abuse or neglect
- Being mostly supported by benefits since birth
- Having a parent with a prison or community sentence
- Having a mother with no formal qualifications
The more of these indicators that a child experiences, the higher the association with poor future outcomes.
For the frontline workers:
We’ve heard from many frontline workers that when you work with people, it’s important to consider what’s going on across multiple aspects of their lives.
Tools like the IDI can help us understand more holistically the situations people face, and the difference you can make.
Why measuring social impact matters to funders and housing providers
Measuring social impact in housing helps funders and providers target services to residents’ real needs, improve wellbeing, reduce costs, and guide smarter decisions across the housing lifecycle with stronger outcomes for people and communities.
Reaching the 15% with the Highest Needs
Reaching the 15% with the Highest Needs – 15% of Kiwis account for half of all social service use, yet mainstream support often falls short. By redesigning services around real needs, we can drive better outcomes for those most at risk.
Trust is the social sector’s most valuable currency
Trust is a vital social currency in the sector. By combining engagement data with lived experiences, organisations can gain deeper insight into relationships and better understand, measure, and strengthen trust with the communities they serve.
The IDI turns 10: Why the world’s watching New Zealand
NZ’s world-leading data system is transforming social investment, helping policymakers and organisations make smarter, evidence-based decisions.
Get in touch
We help impact organisations know, show and grow their social impact. Let’s work together to do good, better!
