Our research and testing methodology
Overview of methods used
The table below provides a summary of the different research methods that were used for each of the users of the service that we identified.
| User | Methods used |
|---|---|
| Short-term let operators and owners | Moderated 1-on-1 session combining a semi structured interview and usability testing Pop-up research for offline journey testing |
| Management Companies | Survey |
| Booking platforms | Semi-structured stakeholder interviews |
| Local authorities | Semi-structured stakeholder interviews |
| Other government departments | Semi-structured stakeholder interviews |
| Related public bodies | Semi-structured stakeholder interviews |
| Bookers | None - no research took place |
| Neighbours | None - no research took place |
Approach to testing with operators and owners
User recruitment
We attempted to recruit operators through a number of different ways. Below is a list of some of the techniques we used.
Recruitment agencies
We used a recruitment agency to help source participants for our testing sessions. Using an agency allowed us to write a detailed brief and try to specify the characteristics that were most important to test our riskiest assumptions. Balancing all of these criteria was not always possible, but using a recruitment agency gave us a good chance to do this. An example of the types of criteria that were in our recruitment brief were:
- Experience (i.e. prospective operators, less than 12 months experience through to 20+ years experience)
- Type of short-term lets owned (i.e. holiday cottages, single rooms, apartments, etc)
- Number of short-term lets owned
- Number of days the short-term let is let out over a calendar year (i.e. from 1-30 days through to 181-365 days)
- Location of short term lets
- Self managed or using a management company
- Places where participants advertise their short-term let (i.e. Facebook, own websites, booking platforms)
- Digital literacy
- Access needs
Direct recruitment
We attempted to source some participants through direct recruitment, for example by posting messages in Facebook groups for owners of short-term lets. This could be a suitable route to find participants in the future, however we did not have the time to manage the responses that came in during this phase of work, and we therefore ceased this approach.
We also found that a number of participants referred us to trade associations and membership groups.
Trade associations and membership groups
We worked with trade associations and membership groups and created a mechanism to capture expressions of interest via a survey. These groups then shared this survey with their members and from this we were able to build a pool of possible participants for research. This panel could be used for future research, however any research and testing that takes place should not exclusively rely on this panel, as not all operators of short-term lets are members of a trade association or membership group.
Testing with ‘offline users’
We attempted a range of routes to identify non-digital short-term let owners and operators in order to test the offline registration journey with them including the paper registration form. The approach taken and findings from these tests are included in ‘Testing the offline journey’.
Engagement
Moderated 1-on-1 session combining a semi-structured interview and usability testing
We conducted 15 sessions with operators and owners of short-term lets. These were split across 3 rounds, with 5 sessions in each round. A summary of the participants that we tested with for these rounds of research can be found in the table below.
| Round 1 | Round 2 | Round 3 | |
|---|---|---|---|
| Experience of owner | 1 × Prospective owner 1 × Less than 12 months 1 × 1-5 years 2 × 10+ years |
2 × Prospective owners 2 × less than 12 months 1 × 10+ years |
1 × 1-5 years 2 × 6-10 years 2 × 10+ years |
| Type of short-term lets owned | 1 × Entire privately owned house(s) 1 × Privately owned self-catering holiday home(s) e.g. Holiday Cottage 1 × Individual rooms or annex within privately owned house(s) & Privately owned self-catering holiday home(s) e.g. Holiday Cottage, Self-catering Apartment, Serviced Apartment 1 × Individual rooms or annex within privately owned house(s) 1 × Entire privately owned house(s) & Self-catering Apartment |
5 × Privately owned self-catering holiday home(s) e.g. Holiday Cottage | 2 × Privately owned self-catering holiday home(s) e.g. Holiday Cottage 2 × Entire privately owned house(s) 1 × Individual rooms or annex within privately owned house(s) |
| Number of short-term lets owned | 1 × 1 short-term let 2 × 2-5 short-term lets 1 × 5 short-term lets 1 × 10+ short-term lets |
2 × 1 short-term let 1 × 2-5 short-term lets 1 × 5-10 short-term lets 1 × 10 short-term lets |
3 × 1 short-term let 2 × 2-5 short-term lets |
| Number of days the short-term let is let out over a calendar year | 1 × let for 31-90 days 3 × let for 181-365 days 1 × Own a property that could be used as a short term let but are currently not using it for this purpose |
3 × Unsure on the exact amount 2 × Own a property that could be used as a short term let but are currently not using it for this purpose |
1 × let for 31-90 days 2 × let for 91-180 days 2 × let for 181-365 days |
| Location of short term lets | 1 × Northwest England 1 × Northeast England 1 × London/Greater London & Southeast 1 × Southeast England 1 × London/Greater London & Southwest |
2 × North Yorkshire 1 × Derbyshire 1 × West Yorkshire 1 × West Yorkshire & Devon |
1 × Southwest England 1 × Northwest England 1 × London 1 × East Midlands 1 × West Midlands |
| Self managed or using a management company | 2 × self managed 1 × use a management company 1 × Part self managed & also uses a management company 1 × None of the above (prospective operator) |
3 × self managed 2 × None of the above (prospective operator) |
3 × self managed 2 × use a management company |
| Places where participants advertise (multiple choice) | 1 × Skyes Cottages 3 × AirBnB 1 × Facebook 1 × Booking.com 2 × Vrbo UK 1 × Holiday Lettings |
1 × Facebook 1 × Airbnb 1 × Booking.com 1 × I don’t advertise online |
5 × Airbnb 4 × Facebook |
| Digital literacy | 5 × 10/10 | 2 × 10/10 3 × 9/10 |
1 × 8/10 1 × 7/10 1 × 6/10 2 × 3/10 |
| Access needs | None | None | 2 × Dyslexia 1 × Dyscalculia 1 × ADHD |
| Assistive Tech used | 1 × iOS speech & Immersive Reader 1 × Isolator & Time Timer 1 × Text-to-Speech & MathTalk 1 × iOS speech, Immersive Reader, & Firefox Reader View |
Analysis
As the outputs of our testing sessions were almost exclusively qualitative, we conducted thematic analysis. We affinity mapped the notes from sessions and were hoping to find patterns, trends, themes from what participants told us (or what we observed them doing) in the sessions.
Our analysis focused on producing findings that would help us:
- learn more about specific assumptions that were being tested in that round of testing
- Make more informed interaction design decisions and allow for iterations to the prototype based on user feedback
- Make more informed service design decisions which would result in an improved experience for our operators who might be using a registration service
- Inform specific questions that had arisen around the policy
How the findings informed design
Our research with operators was conducted in the open and we encouraged observers from the team in every session. As well as live observation, the recordings of sessions were also shared with the team. We also ran collaborative analysis sessions where members of the team could support the analysis process, or be a fly on the wall to see what was happening. We shared findings with the team as quickly as possible to ensure design would not be hamstrung.
Approach to testing with management companies
User recruitment
Direct recruitment
We contacted around 20 management companies directly to invite them to take part in research and testing. We used some sites which scrape booking platforms to help point us in the direction of possible management companies; we then contacted these companies directly to see if they wished to take part. Unfortunately this process did not produce any results and we were unable to book any testing sessions.
Trade associations and membership groups
We used the same expressions of interest survey that we used to recruit owners and operators to help recruit management companies. When we realised we had a dearth of participants in this area, we asked trade associations and membership groups to promote the survey with their members who are management companies. This became our primary recruitment method for these users.
Engagement
Our initial approach was to conduct some rapid research with management companies to try and understand more about them and their needs, feelings and concerns about the registration scheme. However, as our direct recruitment did not provide us with any leads, we had to pivot and try a different approach. This led to us creating a survey to send to the management companies who expressed interest through the trade associations and membership groups. We were able to send this survey to 50 management companies, and we received 16 responses. We decided on this approach as it was the most time efficient way to engage with a large number of management companies. We would have liked to have corroborated what we found from the survey with some interview and testing session however we did not manage to do this in the time we had. We hope to be able to do this in the beta phase.
Analysis
We undertook a mix of quantitative and qualitative analysis on the survey responses. The survey tool used lends itself to detailed data analysis, instead we focussed our analysis on producing insights that would help us:
- Learn more about this user group and their needs, motivations, challenges and worries
- Learn more about specific assumptions which related to management companies
- Make more informed interaction design decisions and allow for iterations to the prototype
- Make more informed service design decisions which would result in an improved experience for management companies who might be using a registration service
- Inform specific questions that had arisen around the policy
How the findings informed design
Findings from analysis were played back to the team over two separate sessions and have led to the creation of personas.