Family measurement tool iterations 1-5

Posted by Sarah on 7 November 2010 | 1 Comments
Tags: Family by Family, measurement, Family Project

At 2pm today we were in the studio quickly mocking up version four of a family measurement tool. By 6pm, we had thrown out much of version four and were working on version five across the Verona family dinning room table. We're in week three of the Family by Family prototype. Prototyping starts with the assumption that most of our ideas won't work, and the best way to learn what might work better is to try over and over again. It's a nerve-wracking and yet exhilarating time. But, how do we know what does and doesn't work? What are our measures of success? Well, on reflection, we use four primary indicators: (1) resonance; (2) engagement; (3) perceived value; and (4) behavioural change.

Resonance is about how attractive the Family by Family message and offer is and to whom. We have an offer for two kinds of families: those that want something to change and those who want to share their experiences of change. While we're not under the illusion that Family by Family will be of interest to all families, we do want to figure out what scripts & materials work best in which contexts and at what times. We want Family by Family to be a movement of families - families who sign-up the vision and values - rather than just a transactional service. We're not there yet.

Engagement is about converting interest in Family by Family into real participation. It's about moving a family from signing-up to coming to a training camp or taking part in a link-up. Here it's about figuring out and pre-empting the barriers between 'saying yes' and 'practicing yes.' This means dealing with practicalities like time, transport and money - as well as giving families a credible sense of the value to be gained by engaging (be that a role, a job, a project, a problem overcome, a new experience, etc.). We're starting to get here.

Perceived value is about making every interaction as meaningful as possible to whole families - it's about the time and space, the look and feel, the language and tone, the activities, the materials, the questions asked, etc. Here, we use our intuition and families' gut reactions: was it a good use of their time? was it surprising? was it unlike anything they've done before? We're just starting to get here. At last week's Wednesday night yarn and dinner, one of the dads noted how his scepticism of Family by Family had been replaced by the value he sees in it for himself, his wife, and his kids. That's exactly what we're going for - something that feels palpably different and good for the whole family. 

 

 

Of course, resonance, engagement, and perceived value matter a lot less if they aren't coupled by behavioural change. We've designed Family by Family to enable more good stuff within and between families. Good stuff is a short hand for families doing, saying, thinking, and feeling more good things, together - or in more academic terms, about collective learning, proactive decision-making, positive feedback, and connectivity. The 'theory of change' that we're testing with Family by Family is that to adopt these behaviours families need to experience and try out these behaviours. Capturing behavioural shifts becomes critical to improving the Family by Family concept, but just as importantly, to helping families maintain and see the rationale for change. That means that measurement must be immediately useful to families. Too often measurement is used for programme evaluation rather than as a motivational tool. We see measurement as a way to make change (or lack thereof) visible and an explicit conversation topic. In other words, we see measurement as less about validating a particular approach and more about co-creating another way forward.

That doesn't mean measurement is easy. On the contrary, it's one of the things we find we have to iterate the most. We are constantly tweaking two elements: (1) the constructs and (2) the modality (i.e. format). This afternoon with the Verona family, we were exploring how to measure 'less bad language' and more 'confidence to go out as a family' using a choice of frequency scales (daily, weekly, etc.). The modality? Magnetic strips for the fridge. It was a bit too complex, although far more focused than our first attempt at measurement six weeks ago. Six weeks ago, the Verona family tested our first measurement tool: a game using poker chips which asked family members to look at how they distributed their time in any given week.

So, what have we learned over the past six weeks about measurement?

Well, first, we've been building off the literature. There are no shortage of family self-assessment tools that have been psychometrically validated, including the Family Assessment Device, the Family Assessment Measure, the Family Adaptability and Cohesion Evaluation Scales, the Family Environment Scale, the Family Functioning Index, the Family APGAR, and the Family Functioning Questionnaire. And this list is only current as of 1991!

I haven't gotten my hands on all of these tools, but from what I've seen, there is a fair amount of overlap in the constructs - family togetherness, connectivity, communication, roles, decision-making, etc. What seems to differentiate many of these tools from the one we're co-creating is their purpose. A lot of the existing tools ask families to rate where they are rather than where they want to be. They are not designed to be ongoing and aspirational. Nor do they disaggregate the constructs by what families do, say, think, and feel. In this respect, these tools require quite a high level of reflective capacity.

With Family by Family, we are hoping to help people develop the reflective capacity to see the link between what they do, say, think, and feel. The relationship between what a family member says and how another family member acts is not always obvious. By tracking changes in how family members talk (say, the amount of positive feedback they give) and how family members act (say, aggressive behaviour) we hope to help families pick up the links for themselves. Behavioural change is always more sustainable when it is self-initiated, and when the reason for that change is backed up by concrete results.  

Of course the devil is always in the details. And we've been particularly bedevilled by the unit of measurement. The first measurement tool looked at the distribution of family time over a single week. Yet we found time was not a great proxy for the quality of interactions, and that for many families, spending more time together was the least desirable thing. It wasn't an aspiration because other things (e.g. aggressive behaviour) were getting in the way. We needed to measure those other things first before families could see any value in spending more time together. Eventually we want families to see the link between variables like time spent together and aggressive behaviour, but that's just not the best starting point.

The second measurement tool we used took time completely out of the equation. We asked families to select which family they were most like: were they the family which was constantly busy and out and about or the family that spent a lot of time hanging out at home, etc? Again, we found that  it was a more descriptive than aspirational tool. Family members didn't immediately see what they could change or why they would want to change. And because of the sheer diversity of families, families found they didn't match any of the cards. The modality we used, a deck of cards, was good for conversation but didn't leave a visible record. 

Our third measurement tool, on a big A3 sheet of paper, did leave a visible record. Here, balance over a single week was the unit of the measurement. Something we learned in our ethnographic work was that thriving families struck a balance between time spent on self and others, and a balance between personality types. These families didn't fit neatly into any one box. Family thriving wasn't about the complete absence of bad language, aggressive behaviour, feeling overwhelmed, reactive decisions, etc. but about the relative focus on rewarding good behaviour, having something to look forward to, etc. So we drew a balance scale and asked families to rate the balance between two opposite constructs - say bad language and good language, etc.

Families liked the third measurement tool. The only problem was that family's had different ideas of what constituted balance, and not all of our constructs had neat opposites. Rather than preset what constituted balance, we wanted to help families figure out their ideal benchmark and calibrate their scales accordingly. Over time, we'd hope to influence the benchmark itself. We had to remind ourselves of our core measurement principle: measurement needs to be something meaningful to families that helps them track progress in the constructs they think are worthwhile. It's not a professional assessment.

 All of these realisations led us to our fourth measurement tool. Here we gave families two cards: one that said 'more of' and one that said 'less of.' Families chose three behaviours they wanted less of, and three behaviours they wanted more of. They could choose these behaviours from a list of things thriving families do, say, think, and feel or they could develop their own indicator. But, rather than set the scales ourselves we let families do so. They could also choose a set of frequency words and calibrate the upper end of the scale with their benchmark number (e.g. saying something nice to each other every day). We would record the distance traveled for families, and conglomerate those numbers at a programmatic level to determine Family by Family's value. This way of doing measurement is far more complex at a programatic level, and requires a lot of supplementary qualitative data to contextualise gains made. But, we felt this would actually be a more accurate indicator of change over time. The only problem was there was too much choice for families - hence the complexity! 

So, there we were today at 6pm across the Verona family table, with dad sketching out another way forward. The Verona family liked the idea that families could choose the indicators that mattered to them, but thought we should standardise the scales. Rather than use a frequency scale, they wanted a perception scale: on a scale of 1 to 10, how concerned were they with the behaviours they wanted less of and how noticeable were the behaviours they wanted more of? We do know from the literature on behaviour change that people's perception of their behaviours is key to taking action.

Of course, we also want families to gain a better relative sense of their behaviours. That's the whole point of exposing them to families that practice a different set of behaviours. Rather than just self-measure, then, we hope the measurement tool can serve as a collective talking point. Sharing families can share their perspectives and impressions and help the families they are working with recalibrate accordingly. In other words, we want the measurement tool to be organic, rather than static. At a programmatic level, we can capture success by how many families shift their target behaviours and how often they are resetting their target behaviours.

The next big question will be, how do we link what family's want to change with the outcomes that governments think are worth changing? We're just starting to assemble an investment case to show why government systems should support Family by Family. Tomorrow, we'll launch a piece of work with Social Ventures Australia to construct a social impact map which ties together changes at a family level to changes at a population and systems level. I believe we'll be able to group the indicators families choose into categories that align with education, health, welfare, and work outcomes. We know shifting patterns of interactions within families can have a profound effect on nearly every other social outcome - we'll just need to take some educated guesses as to how to quantify that effect. Stay tuned.