Day 11. Wireframes
I was mostly flying by the seat of my pants, learning as rapidly as I could. There weren’t designs for anything that the engineers were building. We had a hypothesis of who the users were, and I restructured the site to accomodate the work ahead and iterated with people who were closer to the actual users. Going into this, I knew the wireframes wouldn’t be quite right, but at least they adhered to better practices.
CSS. No Style Guide.
Using the guidelines and colors from CMS, I skinned the site from no CSS. We stuck to Arial because webfonts are typically disabled on their machines. Other considerations were accessibility (Section 508 compliance), dominant use of Internet Explorer, and being within an iframe.
Fig 3. Operations Dashboard with CSS
Validating with Usability Testing
I did some usability testing with a few internal people first, to identify some usability issues and test the script.
Through some magic (we call her Jane), I was able to tagalong to some site visits to do some contextual inquiries and usability testing with actual users. I also supplemented this with remote usability testing with other users.
Download usability script (PDF)
To help designers in the future who need to make things for these state users, I synthesized information from our contextual inquiries into personas.
Fig 4. Carol
Carol triages errors in a state that cares about data quality and has the political and financial resources to improve their data quality. For the past several years, she has worked in a small, technical team and is responsible for all the claims files. Though she feels like she is constantly stuck and cannot get better data, she proactively creates reports (spreadsheets with comments usually) for the people upstream (and to data providers) to highlight the errors out of her control. She also automates a lot of her work to produce better reports, but she relies on querying the database and the Operations Dashboard to ensure that the data they have and send is correct. Carol is impatient to see the files go through and her changes improve their data quality. She is anxious about any change that CMS will decide, and would prefer information about changes months in advance (and then makes notes about each addendum and note). Missing values makes her twitch, because it reminds her of implementing 8/9 fills. She looks forward to when their TA drops bits of information about what other states are doing, and generally feels that they ahead of the curve.
Fig 5. Frank
Frank has been working for a state that is fairly resource-strapped for the past 15 years, but they do want to provide useful information to other stakeholders. Not only does he produce and maintain files for T-MSIS, he is responsible for files for state agencies and other researchers. He remembers which errors he does not need to fix, and efficiently identifies the appropriate errors in the error report that may be a mapping error. He now is able to use the Operations Dashboard to triage errors instead of manually triaging, and sees exactly the data he put in. Though he wants better data quality federally, he can only provide the data from the source, and does not get too worried about the other errors. He wants to see that a code change has gotten rid of the errors that prevents CMS from accepting the data, and is impatient with the Operations Dashboard. Since he is usually doing a lot of things at once, his TA mostly contacts him about any changes, and then Frank sighs knowingly at the lack of forethought. He does make sure that he changes the mapping at the appropriate timing.
Fig 6. Margaret
Margaret does not use the Operations Dashboard to triage errors, since she is a bit more upstream than people who triage. She relies on reports to see if there is anything she can do politically to get people (down to the provider level) to change the data that they are sending to conform to the standard, and prioritizes the technical tasks. She feels like she is stuck in between people that do not care about the data they are providing and CMS, and struggles to provide a clear narrative for the data providers other than that it is federally mandated. As the point of contact for CMS and the TA for her state, she hounds them for information and to provide clarifications her team. Margaret is mostly frustrated with the information lag and lack of control, because she sees her team stall, but has other projects for them to do to improve their data quality. She contacts people at other states to compare notes, and finds that CMS is fairly inconsistent with their communication.
A few key takeaways for the UI:
Download full recommendations including process and documentation takeaways (PDF)
- The interface could provide better information contextual to the type of error that is being shown. Especially confusing are relational errors, which are a new error type.
- It is also lacking in ways to share information, and people cannot share links easily because the UI is within an iframe.
- Known errors keep cropping up, and there should be a way to mute them.
- Users’ mental model is still around errors for each file, even though that isn’t what is happening and can cause confusion.
Below are a few ideas to clean up the usability problems, and presented a large set of wireframes to help with discussions with CMS.
Fig 7. One example of providing more context to a particular error: by highlighting join keys to another record, they can figure out why this record was associated with another record. The plainenglish version of the error is embedded within the record.
Fig 8. One idea to provide relevant, but different information for a value error.
Fig 9. An idea on how to provide different information for a relational erorr (records across multiple files).