MEASURING RELEVANCE

Doron’s story raises a common question about relevance: how do you measure it? If the heart of relevance is people unlocking meaning for themselves, how do you identify the moment when the key slides into the lock successfully? How do you measure something so personal and idiosyncratic? This question is especially complicated when you consider the dazzling variety of keys, doors, and rooms in action.

Answering this measurement question starts with separating relevance at the door and relevance in the room. If an institution is relevant at the door, that means people see the door and choose to walk in. If an institution is relevant inside the room, that means that people access meaning and value through the experiences the institution offers.

Here’s the good news: it is completely achievable to measure relevance at the door. The easiest way to measure the effectiveness of your doors is by tracking attendance. If someone shows up, they clearly made it through the door.

If you care about being relevant to a particular community, then you have to track whether they are walking in the door. It doesn’t matter if a thousand people enter if none of them are from the community to which you are trying to relate. This may require collecting demographic or psychographic data about attendees beyond the fact of their presence. Collecting this data may be awkward. Get over it. Figure it out. If you can’t measure whether the people you care about are coming, you can’t tell if you are moving towards greater relevance to them.

The measurement challenge starts to get trickier once people are in the door. Once people make it in the door, how can you evaluate the relevance of the experience you offer in the room? Repeat visitation is a good starting point. Presumably, if someone chooses to walk through your door more than once, they’re returning because of something meaningful they found in the room. The more often FoodWhat youth showed up at the farm, the more they demonstrated that they were finding value there.

You can also track whether people would recommend the room to others. Some institutions use the “net promoter score” in surveying, asking “on a scale of 1-10, how likely are you to recommend this service/experience/place to someone?” This question helps assess how likely the person is to hold the door open for someone else from their community to come inside. Eager ambassadors and promotores—whether formally designated or informally self-assigned—are often the greatest accelerators of new community participation.

Attendance and recommendations are useful measurements, but they don’t provide much insight into the kinds or depth of meaning that people are building inside your room. Is your role in their mind fixed—as a place to learn, a place to play, a place to take mom when she is in town—or are they accessing different parts of the room at different times? What are they leaving with? If there is new meaning imparted, what will they do with it?

These questions are hard to answer via quantitative surveys at an experience. You can ask people whether they are leaving with a new idea, something they want to go deeper with, or an impulse to make a change in their own lives. But they probably won’t know the answer until after they leave the room—maybe not for months afterwards. If you ask them directly, some people will lie to be nice. There’s no comfortable way to check a box that says, “I got nothing out of this.”

It’s often more effective to identify participants’ experiences through behavioral observation. If you care about whether people are accessing multiple parts of the room, you can track the diversity of activities with which participants engage. If they just come for one type of program, they’re sticking to that part of the room. If they come for different things, they’re navigating different parts—and the institution is likely becoming relevant to a wider set of their interests.

If you care about the depth of meaning that participants derive from the room, you can track the intensity of participants’ affiliation with the institution as a proxy. When participants identify as insiders, they do so because they feel a sense of ownership and deep connection with the room. They don’t just visit it; they are a part of it. And it is a part of them. As abstract as they may see, these feelings are often observable. Insiders exhibit particular behaviors and take quantifiable actions. They adopt the room as their own by becoming members, volunteers, or donors. They fight to change the room—or to protect it from change—by attending public meetings, writing letters to the editor, and providing feedback when asked.

At our museum, we’ve measured depth of affiliation in surveys by asking people directly about feelings of ownership and/or pride in the institution. But more importantly, we observe it in behaviors. People who call us back when we leave messages. Partners who invite us onto advisory boards for community efforts unrelated to the arts. Teen interns who lobby our board of directors for changes to make the institution more welcoming to them. Collaborators who spread the word about the museum to their friends and colleagues.

These behaviors are observable when they happen inside our institution. But that’s not always the case. Here’s where things get really tricky: often, when people unlock meaning in a room, it encourages them to take actions outside its walls. If an audience member walks out of a theater performance quietly determined to be a playwright, she doesn’t keep the theater company informed about her career change. If a trip to a national park inspires a man to spend time outdoors when he gets back home, he doesn’t alert the Park Service whenever he goes for a hike. If a teen in FoodWhat changes his life because of the experience, how would Doron know?

This is the hardest measurement problem. In focused programs like FoodWhat, sometimes staff can track outcomes over time for a small number of alumni. But most organizations don’t have the ability to stay in touch with participants over time. Once in a while, someone will share an anecdote about this kind of deep connection and the way it changed their life. But it’s hard to systematically track the ways institutions ignite new meaning and experiences beyond their walls.

One measurable proxy for this broader impact is reputation. When an individual recommends an institution to a friend, she conveys her perception that value and meaning can be found within. When this happens on a scale of a many individuals in a community—via favorable press, awards, commendations from community leaders, invitations to join boards, solicitations for endorsement—it connotes a broader reputation of value.

That said, it is important to track whether this reputation is exclusive to certain communities. If your institution has a high reputation for providing value among people who are not your target community, that’s irrelevant (and possibly deleterious). If you’re seen as a white institution and you want to unlock meaning for people of color, your reputation will not help you. If you’re seen as a cutting-edge youthful institution and you want to be valued by older patrons, your reputation will not help you. If the people you care about don’t care about the reputation you’ve built, you’re unlikely to be seen as relevant or valuable to them.

How can you measure your potential relevance to people outside your room? It’s less useful than you’d expect to ask them why they don’t partake of your offerings. Remember, for most outsiders, the room is invisible. They don’t know what’s in it, and they don’t know why they don’t come. They just don’t. So if you ask them, they’ll lie. They’ll say something nice, or something misleading. They’ll tell you they don’t come because it costs too much, when their real issue is that they believe your content is boring. They’ll tell you it’s too far away, when their real concern is whether they will feel welcomed there.

Outsiders are not experts about your site or the experiences you offer. They are experts about their own experiences and preferred sites—and that’s the most valuable thing for you to learn more about.

Instead of asking about your institution, ask outsiders about the things they find relevant. Ask about their interests, desires, and decisions. Ask: how do you choose what to do with your free time? Where in our community do you feel most welcome and connected? What’s missing in your community? What issues keep you up at night?

When you listen to their answers, you may be surprised to find connections you didn’t know were there—doors that would be easy to frame out. You may also find things that challenge you—and make you question whether it is possible to build a door for those people at all.

I once facilitated a workshop at the State Library of Queensland. The library was building a new center for indigenous knowledge sharing. As a public institution, they felt it was important to be a place for all kinds of knowledge in the region—not just information fit for shelves.

The library was proud—rightly so—that they had apportioned out an area inside the library to be dedicated as an indigenous knowledge center. We sat down with indigenous culture-bearers at this early meeting. We asked: how do you share knowledge? The indigenous elders told us: we pass it down in stories, told orally, from generation to generation. Fine. Check. We could have a story-sharing area, maybe even a way to capture those oral histories digitally to be shared over time.

Then the elders told us: we tell the stories with music and song. We tell the stories outside. We tell the stories around a fire.

The librarians looked around nervously. They weren’t prepared to offer the kind of environment that would be truly relevant to indigenous knowledge sharing. To their credit, they identified a potential site outdoors for music, though they continued to grapple with the question of whether fire was possible or desirable. Eventually, they decided it was an essential component to add. Now, the library boasts an outdoor knowledge-sharing area called the Talking Circle, complete with fire pit.

When you ask outsiders what is relevant to them, you don’t just learn what matters to them. You learn what matters to you, what protocols you are able or willing to shift and which you are not. And that’s a third way to measure relevance: by measuring your own institutional tolerance for relevant activities that challenge traditional ways of working. Are you willing to rebuild your room to be relevant to outsiders of interest? That’s what the next chapter is all about.