Stage 5: Testing the prototype

Here we collect notes from testing and reflections.

Script for testing

Preamble/context…

  • background information
    • governance design process trial!
    • user/group participation
    • involving people from groups in Gothenberg and Stockholm, academics, and me…
    • improving governance features of karrot
    • lots of notes on the forum…
  • brief explanation
    • prototype, not the real thing, doesn’t connect to other users, only in their browser/computer
    • it’s not a test of them, but of the feature
    • 3 scenarios to go through
    • one person to lead the session, the others are observers
    • afterwards some chat!
  • instructions to think out loud as they go through it

https://karrot-prototyping.netlify.app

Propose an existing document

  • Start with a written document that you want to use within your group
  • On the home page press “Reset to empty data”
  • Navigate to the Agreements tab
  • Make a proposal for the group to accept the document

Participate in an existing discussion

  • Go back to the home page
  • On the home page press “Reset to sample data”
  • Navigate to the Agreements tab
  • Select an existing proposal
  • Make a comment in the discussion
  • Cast your vote

Change an existing agreement

  • From the Agreements tab
  • Select an existing agreement
  • Read the agreement
  • Propose a change

Notes from the test of the testing of the prototype with a participant from Solikyl (gotemburg)

How do we record? Dropbox account? Jitsi?
We need sth that records both voice and screen

General information
The little tasks
Talk outloud as you do stuff
it took us an hour and a half

Suggest an agreement/make a proposal

Go back to check how other agreements look like as an example → example agreement

How do tags work? Is it for a better search?
tags do not always read good

We missed one bit → timeline for agreement
Make it more clear that is a proposal for an agreement
Become a proposal → surprising…
Date doesn’t come back when your editing a proposal
Plus button can be misleading
Could be different → established agreement

Participate in an existing proposal
Negative voting can’t see the rest of the text
He didnt get the connection with the text box (chat box)
Write a message doesnt sound ok → does not promt you to make a comment

Propose a change to an existing agreement

Text with values doesn’t read good actually
Edit proposal vs suggest changes → can you change the title?
Istead of making an edit participant wrote my proposal is
I wouldnt overwrite someones text?
Who has the ownership of a proposal?
Voting control?
Not participating in the chat? →
Question using a proposal? Should solikyl take money?

notes

jitsi recording works good (both voice and sharescreen)
recs are save in a dropbox account (can we have a shared dropbox?)
agreements tab: can we improve it with some text when it’s an empty screen? make it more intuitive, a better explanation
proposals and agreements are the same thing when there are no data…so maybe we only need the proposals tab and option
established agreements? → no discourse how this works? how we support it with the software?

Reflections

  • importing initial established agreements, not clear that it is just making a proposal
    • hence didn’t notice the time period part, it was just meant to be an initial “import"
  • wondering the purpose of the value tags?
    • searching…?
    • for us the text is clear explaining what the reason is…
  • when making a proposal, wrote a comment inside the text instead of editing it
    • “who am I to change someones view” >(e.g. why/how is how allowed to edit someone else’s text)
    • wondering why/if the comment could go in the chatbox… didn’t make so much sense
      • could work, but not intuitive
      • maybe just a issue with our framing? editing vs commenting?
        • google has editing, viewing, commenting modes (I think)
  • chat box covers some of the screen
    • should be fixed!
  • confusion between agreements and proposals
    • creating agreement and creating a proposal is actually the same
    • if there are no agreements, have a button to propose one
    • maybe making it that proposals are kind of a “sub part” of agreements and inside the agreements list UI
    • maybe colour coding, and making clear there are “approved” agreements and “proposed” ones
    • maybe mirrors members + applications, processes that lead to things (application → member, proposal → agreement)

Notes from test with a member of Stockholm group

22/04

#Task 1: Create a proposal from a sample

  • Instructions are better given spoken, not written
  • instructions weren’t updated since there is not a “proposals” page now
  • “I wonder if I change the time period”
  • icons in datepicker took a while to load
  • Some confusion regarding the task itself (make up a sample, or based on real-life examples?)
  • rules vs agreements - “where do rules fit into this page/form”
  • “saw the values button and just checking it out”, “I wonder if I can click these” … possibly wondered why clicking one value highlighted it multiple times? (or maybe just me…)
  • “I get I’m supposed to submit a proposal and getting feedback from the group”, but not sure about rules vs agreement part
  • “I think it might be really useful to have, it keeps on happening on karrot, there is something to be discussed, and lots of threads, but not easy to come to some sort of conclusion or agreement, so makes sense”

#Task 2: Participate in an existing proposal

  • test setup: maybe clearer language for setting scenario data
  • imaginging the list of values is things that the group has already agreed on, or whether they are from karrot itself (made more sense to him that they would have come from the group)
  • skipped the step to propose a change… then confused where to make a comment … ah it’s the difference between an existing proposal vs an approved agreement
  • found the proposals eventually :slight_smile: “just a matter of getting used to it, if somebody tried it once it should be clear, wasn’t initially sure on the difference between the tabs” (approved/agreements)
  • double checked that he had done the voting, checking the title of the voting section

#Task 3: Change an existing agreement

  • found the “propose a change” quickly here, and it made more sense from the previous task confusion
  • wasn’t clear that it went into the “proposals” list… looked in the approved agreements section

Overall / general impressions

  • based on current stuff in the group, they need to come to a decision about how to deal with a particular supermarket, it’s currently very unclear in many threads, no time limit… can immediately see a direct use for it. decisions that need to be taken. currently have conversations within karrot and seperate groups.
  • rules vs agreements? not clear on what rules would be in the context. a standard set of rules, or something over time? when somebody joins, something they should read? softer word than rules could be guidelines
  • good idea for people to join can have a say, but maybe for new people they could spend some time to get familiar with the group first. maybe the karrot trust system is good at filtering people who are active, so better for people who know a bit know about the situation to have a say, of course comments are welcome. for voting it would be nice and inclusive to have newcomers participate in discussion and voting. good to make people feel included.
  • to get people involved, making it clear how it works is important, maybe with a notification about the feature, and to invite them to have their opinions heard, from experience, in general people respond to polls much quicker than discussions which are more unclear. but yeah, understanding how it works is important.
  • never ending discussions :slight_smile: that have one thread, and then maybe another post/thread, seems nice idea to handle the discussions separately that need to reach some sort of agrement. currently, if people aren’t subscribed to a thread they won’t see the discussion happening.
  • values feature? was searching for rules at that time, so maybe got confused there. ponders about values. wondered where the set of values come from, the group? do they set them up, then after that can link them into the agreements. or a standard exhaustive set that karrot provides that people can then connect to that specific agreement. don’t have a strong opinion for giving feedback on it. ponder ponder. was quite clear what it meant, clearly under two headlines, what causes is this relevant for. lack of clarity was about trying to find the rules :slight_smile: … and confusion about who decides the list. seems useful to link discussions to the values the group has. maybe could be more flexible having custom ones.
  • confusion of agreements/proposals. saw the approved agreements that looked like they could be proposals, but maybe just because trying to do a task. but understand the two terms afterwards. at the beginning didn’t have clarity around the proposals vs approved agreements (maybe we can put explanation in the ui). “propose a new agreement” → wouldn’t that be a proposal? although didn’t take too much time to get used to the meanings.

Notes from user testing session with participant from Foodsharing Warsaw

Date: 2021-04-29

  • other feedback…
    • yesterday their meeting they have every 5 weeks
    • they discussed for half an hour why people post on facebook instead of karrot
      • unfair, as everyone is on karrot, but not facebook
      • it’s quicker on facebook, better for reactions, in a hurry…
      • maybe if there would be features on the wall? better? more developed
      • communication features more refined it might help, eaier, more effecient, clearer, more organised
      • could switch to use karrot for more
      • sometime end up having the situation where they agree things on facebook
      • easier to discuss on facebook as people are used to the functions
      • can you delete entries on the wall in karrot
    • idea to invite more feedback from groups, multiple people from groups, have discuss, get feedback, etc…

Propose agreement from existing document

  • started reflection on the discussion they had in their group and that they wanted to propose a new rule
  • “Looks weird that approved is on the left and proposals on the right”. Was expected the other way around
  • term could be cleared
  • looked at the time scale (7 days). Talked about the conflict resolution, 7 days is too long.
    • for a proposal is ok
    • for a conflict is too long, too much conversation and wait long for a decision
    • chose 5 days
    • used the reason for proposal (short version, normally would write more)
  • “values? ok… what values?”
    - “I wouldn’t use it” see what you mean
    - “it would be difficult, I’d spend a really long time to decide what it is and not sure if it’d be useful. Maybe it would.”
  • Not sure whether reason and summary would be separate. One would be enough
    • start the summary with the same thing “we discussed and agreed, etc”
  • voting
    • not clear, assumes voting will happen
    • sees the propose page and the voting voting
    • what is enough people, regarding number of votes
      • is it % of users? how does it relate to trust system?
      • Read the chat bubble about voting negatively
        • “ah ok, seems cool/intersting”
        • Wrote something on the chat
        • maybe people would write whatever just to vote negatively
        • got no feedback from voting like “Are you really sure?”
          • Would like to have a confirmation
          • She sees however that it can be changed
          • Wait for others to vote and see what happens after 5 days

Participate in existing discussion

  • looked at tabs
    • not sure what approved is…
    • Seems like archive of old stuff, but also “coop with companies” that is under proposals
    • approved through vote or just put there?
    • maybe active ones should be first
  • looking at discussion
    • opened the full document: Woaaah, long thing
  • show changes
    • “ok, I don’t know… hmmm, ok…”"
    • not sure what it was
  • edit
    • can edit even though did not create it
  • prone to errors
    • Karrot is not hierarchical, works most of the time
    • peopple could do a mistake when editing
    • not clear whether people can vote and edit
  • forgot to make a comment
    • commented the text on the chat
  • edits the proposal to test it (would not do it otherwise, but ok in the context of the test)
    • checked for changes and saw that her change was included
    • “this is not clear to me”
    • I discovered something. Normally would not like to make an error/edit someone’s work
      • I’d like to comment, not like to edit

Change existing agreement

  • You said “propose a change”…
    • oh this has values, I thought the other one did not have
    • confused about agreement on approved and proposal with the same name “cooperating with stores”
  • Reading the agreement, clicked propose a change
    • Maybe someone could have abused it and set for one day
    • 7 days default is too long
  • Reason for proposal is provbably reason for change. Or is it reason for rules on meetings?
    • “I want to change point 6 because of…”. Assumed reason for change
    • I’m confused, repeated the above
  • Change point 6
  • Checked changes
    • So there is a place where people can see changes
  • Already I can vote on my own suggestion
    • voted “of course I support it, my own suggestion :)”
    • probably commented (screen froze again)

Chat after testing

  • Can it be useful for her group?
    • Needs to think
    • first move all the rules to Karrot which is good (everyone has Karrot, not Facebook)
    • now they have a link on both Karrot and Facebook to google docs
    • difficult to know how people would be using it
      • maybe everyone would be using it a lot proposing a lot of changes
      • but from experience, that would rarely be the case, a few people more into organizing and coordinating
      • 2 kinds of meeting: everyone and org. group
    • experience from yesterday:
      • they had the big meeting and had a clear idea of the rule, but it would take time to do it together, but the smaller group would take care of it
      • expects that it would work similar in practice with the feature
      • in the end it’s like a meeting, but online
      • everybody can take part, would be an advantage
    • could it increase participation for less active users?
      • hard to say… only have experience from the conflict resolution feature, don’t use it for all, as the only option is to throw them out or not… but they do start them in that case, 100 comments in 3 days!
      • not sure if non-coordinate people would use it, at least how it looks, as it’s quite complex looking, people might be intimidated, people who things a point should be changed, would rather write a comment on the wall, they would put it as topic for next meeting
    • something in karrot, softer sanctions
    • already have processes for making agreements,

Jitsi froze screenshare 3 times

Our reflection

  • confusion about the approved/proposals left/right thing
  • confusion when there is a proposal for changing existing one as they have the same title
  • fields for “reason for proposal”
    • similar to summary?
    • reason for the agreement itself or reason for change?
    • she had the reason, maybe just changing the name would make it clear
    • and remove “reason for proposal” for new proposals?
    • putting “(optional)” might make people for more comfortable to leave it
    • clearer distinction between meta content (duration, reason) and the actual proposal (or “agreement”)
  • social norm of “changing someone elses work” echoing joakim
    • in context of modifying an existing proposal
    • prefer to make a comment
    • respecting the persons authorship
    • should we design for a collaborative process or adapt to these social norms?
  • question about active/less active people in the group
    • her remark: people less active would probably feel more comfortable to write on the wall instead of on the agreement
    • where do discussions take place?
    • should we have a discussion under approved agreements?
    • similar to Github’s issues and PRs
  • pulling in existing agreements without re-debating/re-voting
  • they already use google doc, do they use the commenting feature already?
    • she didn’t mention it
  • values!
    • phew, would take a lot of time to think, and maybe not too useful
    • if we had this idea where the group has already select it’s values, then each proposal is a subset of the group values
    • maybe keeping values to the group vision/governance stuff makes sense now?
    • group description (to be renamed to information) is also related, as that’s where people put things about the group, could include visions, etc…

Facilitator for next meeting: Bruno

1 Like