Developing Annual Surveys for Data Visualization Style Guides

When working in an organization with visualization needs, maintaining an effective style guide is crucial to keeping consistent styles and speeding up the work flow. But how do we measure style guide efficacy? 

One solution is through the strategic use of annual internal surveys. These tools not only assess how well a style guide meets its objectives, but they also reveal areas that need improvement and the guide’s overall impact on an organization. This article will delve into the importance of these surveys, explain the role of leading and lagging indicators in this context to help frame your survey questions, and provide actionable steps to design and implement such surveys effectively.

Implement a regular schedule (every six months or so) for conducting these surveys to ensure you consistently collect data for analysis and comparison.

Understanding and implementing lagging and leading indicators

Lagging indicators

Lagging indicators, such as user satisfaction rates or the decline in errors from the application of the style guide, can give a clear picture of the guide’s effectiveness. They provide tangible, quantifiable measures of how well the guide is working after it has been implemented. Lagging indicators are measurable factors that change after executing business processes or strategies. Examples include increased revenue, reduced costs, enhanced brand reputation, or improved customer satisfaction. These indicators reflect the business’ past performance and are useful for confirming long-term trends. Benefits include:

Justifying the style guide’s existence

If the style guide contributes to positive lagging indicators, such as increased productivity or improved data accuracy, they can be used to justify the guide’s continued use and development. Resources for this work are always scarce, so it’s important to quantify the return on investment.

Setting goals

Lagging indicators can be used to establish benchmarks for future performance and set goals for continuous improvement. For example, if a lagging indicator shows that the style guide has reduced data visualization errors by 20% in the first year, a goal may be set to reduce errors by another 10% in the second year. In this context, ‘errors’ could refer to problems in creating the visualization, such as inconsistencies in color schemes, typography, or chart types, or they could be issues in interpreting the viz, implying a lack of clarity or intuitiveness in the design. Admittedly, this can be very difficult to measure. Regular audits of dataviz work can help identify and quantify these errors. User feedback surveys can also serve as a valuable tool to gauge satisfaction and identify areas for improvement. One further possibility could be comparing visualizations created with your style guide to those created without it. This can help you determine whether your style guide actually improves the quality of your visualizations.

Regular audits of dataviz work can help identify and quantify errors. User feedback surveys can also serve as a valuable tool to gauge satisfaction and identify areas for improvement.

If the lagging indicators do not show improvement, it may indicate that the style guide is not effective and needs to be revised. For instance, if the guide’s user satisfaction rates are not improving, it could suggest that the guide is not user-friendly or not addressing the users’ needs.

Remember, while lagging indicators are important, they only show the outcome of past actions. They should be used together with leading indicators, which can predict future performance, for a more comprehensive evaluation of the style guide’s effectiveness. 

Identify the lagging indicators for your organization and analyze how they relate to the effectiveness of your dataviz style guide.

Leading indicators

Leading indicators are measures that can predict future performance or outcomes. They are called “leading” because they change before the process or trend they are tracking changes, providing an early warning signal. They serve as a bridge connecting the style guide to business objectives represented by lagging indicators. These metrics could be the number of downloads of the style guide, website traffic, user satisfaction, and others. They provide a real-time assessment of your style guide’s performance before the lagging indicators can be measured, helping you make timely adjustments. When it comes to a data visualization style guide, leading indicators can help foresee its usage and effectiveness. Here’s how you can evaluate these:

Training completion

Establish a way for employees to learn about the style guide. This can be through self-training modules or by attending presentations that cover the contents of the guide. Track the number of users who have completed this kind of training. This metric can be a leading indicator of future usage as those who are trained are more likely to use the guide.

Engagement with the guide

Monitor how many users are actively engaging with the guide, such as asking questions, making suggestions, or providing feedback. High engagement can be a leading indicator of future usage and effectiveness.

Adoption rate

Measure how many users start using the style guide after it’s introduced. A high adoption rate can indicate the guide will be effective as it’s being utilized.

Usage in projects

Track how many projects or tasks incorporate the style guide from the start by completing an audit. This can indicate how much users trust the guide and its potential effectiveness.

Satisfaction with training

Conduct surveys after training sessions to gauge user satisfaction. High satisfaction can lead to increased usage and effectiveness of the guide.

Time spent on the guide

Monitor how much time users spend on the style guide’s web page or document. More time spent could indicate users are trying to understand the guide, which could lead to future usage.

Ease of use ratings

Ask users to rate the guide’s ease of use. This can be a leading indicator of future usage and effectiveness, as users are more likely to use something they find easy to use.

If you’re unable to measure these leading indicators because, for example, your organization is not offering formal training on your style guide, this might be a good opportunity to build such a training into the organization’s process.

Remember, leading indicators are predictive and not definitive. They should be used in tandem with lagging indicators to provide a comprehensive understanding of the style guide’s usage and effectiveness. Consider the tools at your fingertips, your time, and importance of each of these leading factors to your goals in order to determine how to track leading factors.

Determine the relevant leading indicators that tie your style guide to your business objectives and track them regularly.

Surveying your users

At the end of the day, the most direct and densely informational feedback you will get is through user surveys. 

Formulating survey questions

Creating effective survey questions is a crucial step in measuring the style guide’s impact. The questions should be clear, unbiased, and focused, designed to elicit responses that would help to either prove or disprove your hypothesis. They should cover areas such as ease of use, helpfulness, and impact on productivity.

Start by mapping out what the organization values (e.g., increased revenue, reduced cost savings, stronger brand, etc.) including how those values are quantified. These are your lagging indicators. Then create a hypothesis that connects the style guide to these figures via intermediate metrics (e.g., downloads of the guide, webpage traffic, user satisfaction, etc.).

Now create survey questions that will help to disprove your hypothesis.

Example questions:

Does our style guide empower employees to be successful?

• To what extent have our style guide’s updates and new additions from the last year made your work harder or easier?
• How easy is it to find the resources you need on our style guide’s websites?
• How difficult or easy is it to learn to use our style guide?
• To what extent does our style guide help you be less or more productive with your work?
• How often do our style guide’s websites contain the resources you need for your work?

Does our style guide have a positive impact on employee culture?

• How many contributions have you made to our style guide?
• To what extent has our style guide influenced your satisfaction with your role at [company]?
• To what extent has our style guide impacted your satisfaction with [company], in general?

Does our style guide help employees form good habits?

• How much do you trust our style guide’s guidelines?
• How clear are the guidelines for our style guide’s design principles and components?
• How has our style guide impacted the quality of collaboration between designers and engineers?
• How satisfied are you with our style guide?

A survey should also collect demographic information like job title, department, length of time at the company, how often they use the guidelines, and if they were trained on using the guidelines. This information can help you make connections between how people act and how they feel. It will help you understand your guideline’s shortcomings. It will also help you understand the value the guidelines bring to the organization, which is an important step in securing resources to maintain and build the guidelines.

In terms of survey avenues, while many resources are available in terms of survey tools (Google Form, Qualtrics, Survey Monkey etc.), qualitative surveys — such as one-to-one interviews — can sometimes be more useful than a written survey. Mixing the two is a good compromise. At the end of the day, the survey tactic you use depends on your goals, time, and overall capacity for the project. 

Analyzing and interpreting survey results

Analyzing and interpreting survey responses can be a complex process, but the insights gained can provide valuable guidance for improving a data visualization style guide. Here’s a step-by-step guide on how to do it:

1. Data Cleaning: Start by cleaning your survey data to ensure accuracy. Remove any incomplete or irrelevant responses.
2. Quantitative Analysis: For closed-ended questions, calculate the frequencies or averages of the responses. This will give you a general idea of the trends in the data. For example, if most respondents rate ease of use highly, your style guide is likely user-friendly.
3. Qualitative Analysis: For open-ended questions, categorize the responses into themes. For example, if multiple respondents suggest adding more examples in the guide, this forms a theme you can address. Using mind map tools such as miro or figjam can be helpful in organizing themes.
4. Cross-tabulation: This involves comparing the responses of different demographic groups. For example, if newer employees struggle more with the style guide than their more senior counterparts, you may need to improve onboarding materials.
5. Benchmarking: Compare your results with previous surveys (if any). This can help identify areas where your guide is improving or lagging behind.
6. Interpretation: Interpret the results in the context of your objectives and hypotheses. If a significant number of respondents don’t find the guide helpful, but your hypothesis assumed they would, you need to reassess your guide.
7. Action Plan: Based on your analysis, create an action plan to address the feedback. This could involve updating sections of the guide, providing additional training, or even redesigning certain aspects of the guide.
8. Communication: Share the results and your action plan with relevant stakeholders. This could be the team responsible for the guide, the management, or even all the employees in the case of a company-wide style guide.

Remember, the goal of the survey is to gather actionable insights to improve your style guide. Every step, from designing the right questions to interpreting the results, should be conducted with this goal in mind.

Foreseeing challenges

Conducting these surveys isn’t without challenges. Your first survey will be the most difficult and least valuable because it only provides a view of one moment in time. Future surveys will be easier (because you’ve done all the work to create it) and more useful because you’ll know the direction your metrics are trending. Other difficulties you might face could be in formulating the survey questions or encouraging participation among employees. However, you can overcome such hurdles by ensuring the survey is easy to understand and complete, providing incentives for participation, and communicating the survey’s importance. You might consider making some questions optional, putting time on the user’s calendar for the survey, and conducting a few 1-on-1 interviews.

Regularly review your survey processes, identify any challenges, and strategize on ways to overcome them to ensure your surveys remain effective and beneficial to your style guide evaluation.

Closing thoughts

Surveys are the heart of evaluating the effectiveness of your style guide. Understanding and effectively using both lagging and leading indicators is crucial in evaluating the effectiveness and predicting the future usage of a data visualization style guide. Lagging indicators, such as user satisfaction rates and error reductions, provide a clear picture of past performance, while leading indicators like training completion rates and engagement levels can help foresee future trends.

Conducting regular surveys and tracking these indicators allows for a continuous feedback loop, ensuring that the style guide remains relevant, user-friendly, and beneficial to the organization’s goals. It’s important to remember that these indicators should not be viewed in isolation but used in tandem to provide a comprehensive overview of the guide’s impact.

Through regular evaluation and adjustments based on these metrics, a data visualization style guide can continue to evolve, driving improved data comprehension, decision-making, and overall business performance. Surveys even have the added bonus of providing more awareness of your guide and the resources available. Always remember: The goal is to create a style guide that not only serves its users effectively but also contributes positively to your organization’s broader objectives.

For more resources and further reading, check out https://www.datavizstyleguide.com/.