Custom Form Builder

[kasugaijin]

Background:
Each organization needs to know information about new adopters and fosterers who register. Often, the questions between organizations will differ, so an organization needs to be able to define its own questions.

The two user types that are asked questions after creating an account are adopters and fosterers - the questionnaire for each may overlap, but also be different. A user can also be both an adopter and fosterer. The responses to these form(s) make up their AdopterFosterProfile that is used by staff.

Custom Form Builder:

• Each organization's staff can CRUD their own questions, defining the question and data-type.
• Questions can be grouped into questionnaires (e.g., for new adopters, or new fosterers).
• The adopter Questionnaire may overlap with the fosterer questionnaire. If a user already has responses for a question (e.g., an adopter also becomes a fosterer) these fields should be pre-populated so they do not need answering again.
• Responses to questions should be readily retrievable to populate the adopter/foster’s AdopterFosterProfile.
• Phase I - developers can CRUD questions and create a questionnaire (a grouping of questions)
• Phase II - integrate with the front-end so staff can CRUD questions/group into questionnaires, and new adopter/fosterers are shown an org’s questionnaire after account creation and the responses populate their AdopterFosterProfile

[uoodsq]

Data Modeling

Here's one possible approach to modeling this. I tried to take into account the context here and in the discussion in #388.

Edit: Changed Form -> Questionnaire and type -> account_type based on discussions.

Questionnaire

The Questionnaire model encapsulates a group of questions around a specific topic (e.g. lifestyle, housing situation, etc). It has a title and optional instructions defined by staff users that are displayed to adopter/fosterer users.

Questionnaires have a many-to-many relationship with Organizations using the OrganizationQuestionnaire model. The join table specifies a few additional properties, including the type of join (e.g. adopter, fosterer), and the order in which the questionnaire should appear when viewing/editing a profile. This relationship should enable this functionality:

The adopter Questionnaire may overlap with the fosterer questionnaire.

Question

The Question model is similar to what was originally proposed. It has an input_type which'll be an enum in the app that maps similarly to Google Forms question types (short answer, long answer, choices, etc). It has an options field that stores arbitrary settings that are specific to input_type, like choices, placeholder text, and so on.

Like Questionnaires, Questions also have a sort_order which drives the order in which questions are presented within a profile.

Answer

The Answer model stores an answer to a question as provided by a User. I messed around a bit with Response and Profile classes that encapsulated groups of Answers but this bit stuck out to me:

If a user already has responses for a question (e.g., an adopter also becomes a fosterer) these fields should be pre-populated so they do not need answering again.

If we are having to map answers to Responses/AdopterAccounts/FostererAccounts I think there's more risk of ending up in a situation where we are missing FKs and leaving already answered questions out of certain profile types.

Continuity

I read up on some of the discussion around how to deal with updating and removing questions. Personally, I would prefer to use soft deletes and gently warning staff users about changing the meaning of questions, but I do also understand that users may not always use the app the way we intend. Some well-placed frontend warnings can help mitigate people changing the meaning of questions, but there's also no harm in just having another field on the Answer table for dumping what the question was at the time it was answered by a user.

[uoodsq]

If I'm reading this thread correctly, it sounds like the data model being proposed here should work. I'm not worried about settling on naming conventions here, and I'm going to start putting together some wireframes for how the UX of this should work which should give us something more concrete to focus on as to how admins and users will interact with this. :)

[kasugaijin]

Yes I think so, @uoodsq !

[elight]

New guy here with some possibly naive questions and a few thoughts, after looking some at this schema and schema.rb.

About this schema: how would this represent a list of people who filled out a given Questionnaire for an Org? It looks like it'd have to be a multi-way join of User -> Answer -> Question -> Questionnaire -> OrgQuestionnaire?

Should OrgQuestionnaire belong to AdopterApplication?

At the risk of over-stepping, I wonder about the unity utility of re-using individual questions, within an organization. What is the use case for this? I understand that there will be some re-use between adopter and fosterer questionnaires. Would it be easier, from a UX and modeling perspective, to offer an abstraction that bundles sets of questions together such that one could drop in a bank of reused questions, such as "Previous Pet Ownership Experiences" which could contain multiple questions?

[kasugaijin]

A PR was merged for the building blocks: #629 with some renaming
I think this approach gives more flexibility for an org to define and group questions together?

[uoodsq]

@elight Great questions, thanks. As @kasugaijin mentioned, names of things ended up changing in what got merged.

how would this represent a list of people who filled out a given Questionnaire for an Org?

I suspect the query to generate this list of people would be pretty gnarly, or it'd end up being something that got assembled Ruby-side. This particular use case hasn't come up yet, nor was performance a target, so I opted to try to keep the schema on the simpler side (whether or not that ended up happening is debatable 😆)

The use case I had more in mind was for rendering individual profiles, so given a specific user and profile type, iterate over the Forms and then the Questions, rendering either the Question as a form input, or loading the user's Answer and displaying it.

Would it be easier, from a UX and modeling perspective, to offer an abstraction that bundles sets of questions together such that one could drop in a bank of reused questions, such as "Previous Pet Ownership Experiences" which could contain multiple questions?

Yes! I think #629 addresses this. Form encapsulates a group of questions like you describe. FormProfile "attaches" the Form to a given profile type ("adopter" or "fosterer") with a sort order.

[kasugaijin]

So the app is purely in development. I think we can get rid of any redundant tables without any need for handling existing data.

[uoodsq]

Sounds good. We could still seed new Organizations with some boilerplate questions.

[kasugaijin]

Yes you are right. We don't necessarily need a new table. Perhaps there may be value in some class that collates this data for a given user? But it can pull from the Answers table without any issues I think. So, in that sense, the AdopterFosterProfiles table would probably become redundant.

[uoodsq]

To me, adding that type of table feels like something that should be done if the app can't express what it needs out of the tables with all of what ActiveRecord has to offer in a performant manner.

[maebeale]

Agree! Esp since the same humans can be both. Going to a User Profile, seeing the answers they have can be grouped by FormSection/Form/QuestionFrom/FormQuestion in the display. And if really we only want to see a particular one at a time, we can use a url param from whatever they came to hide/show the parts we want visible for that purpose.

[uoodsq]

FormSection/Form/QuestionFrom/FormQuestion

🤣 names are hard

[nsiwnf]

Was looking at the Baja Pet Rescue adopter form - is it worth building in some "conditional" logic into questions?
Ex.
"Will dog ownership be regularly shared?" Yes/No
If Yes -> "Briefly describe who else will share dog ownership"

[kasugaijin]

Yea this is a good question. I think it would add value to have the ability for a question to show or hide depending on the answer to its parent question that it belongs to.

Thinking about it quickly, to facilitate this we would have to enable staff to create a question that belongs to another ie the parent and child. In practical terms, the parent would be a Boolean, or maybe a select, and we could use Hotwire to render the child question depending on the selection made on the parent question. Does this pass the sniff test?

[uoodsq]

I worry that modeling dependencies between questions could get messy. What if you encapsulated the boolean and details fields in a single Question with a custom input type? (e.g. "Yes/No with details")

You could still ship whatever frontend bits are needed to switch the fields on/off, but the answers would get stored in a single Answer row.

[kasugaijin]

Ah yes this makes far more sense, I prefer that approach @uoodsq

[uoodsq]

I've put together some wireframes to illustrate the experience for staff managing forms and questions:

A new section on the dashboard sidebar ("Forms") takes you to a page where you can see all Forms that exist for a given organization. The table indicates how a Form is used (Adopter profiles, Fosterer profiles, or both):

Adding/editing a Form, including selecting the profiles to which the Form applies:

Once a Form is created, it can be visited to see the Questions that exist in it:

Adding a new Question:

Changing a Question's type drives the options that are presented to the staff user. For example, changing to multiple choice allows the user to enter the choices presented to the end user:

Once Questions are added, a separate page exists that allows the staff user to drag/drop and reorder the Questions within the Form:

Similarly, Forms can be reordered within a profile type (adopter, fosterer):

[kasugaijin]

This is awesome, thank you @uoodsq
We had our project weekly tonight and went over what you have provided, and it all makes sense. I for one am happy to see how it might look, and it all looks very straightforward and logical to me.

We had a couple of discussion points come up:

• Editing/deleting questions by org staff - in the discussion above we agreed that org staff should be able to edit questions, but provide a warning that meaning should not be changed. What should we do if org staff delete a question? I agree with your proposal to soft delete answers. Do we also want to save the question text on the response to provide contextm in case a question is edited (and inadvertantly meaning changes)?
• If an adopter or fosterer user completes their profile questionnaires, and then staff add another question to a form, or form of questions at a later date, the adopter or fosterer will need to be notified they have questions to complete.
• How does drag and drop work on smart device/touch screen? I have not worked in drag an drop in rails, nor for smart devices.
• Is there a necessity for a ‘required’ form? My thinking is no. A form is something all users will need to answer (save for the user specific adopter/foster form which only each user type answers).
• number of forms - so each response to a form question will be on the same model right? So is it feasible to have a separate form/submit for each form? Ie a user can just fill out the housing form and submit it? On a related point, how can we handle validations? I am not quite understanding if we can have back end model validations or not. I am used to using them when we know the form fields and attributes being used. But in this case we don’t know them ahead of time. This also relates to the separate form submissions, if we submit only part of a models attributes will validations for other attributes fail? I am sure there’s ways to solve for this but it’s new to me :)
• to @nsiwnf point above, some questions will have associated questions that e.g., anserwing Yes to 'do you have pets at home' would then show its related question of 'describe what and how many'.

[uoodsq]

Editing/deleting questions by org staff - in the discussion above we agreed that org staff should be able to edit questions, but provide a warning that meaning should not be changed. What should we do if org staff delete a question? I agree with your proposal to soft delete answers. Do we also want to save the question text on the response to provide contextm in case a question is edited (and inadvertantly meaning changes)?

Yep, I think soft deletes + a question_snapshot on the Answer model will be a good way of capturing what the state of the question was at the time the answer was made without breaking the logical relationship between the question and answer.

If an adopter or fosterer user completes their profile questionnaires, and then staff add another question to a form, or form of questions at a later date, the adopter or fosterer will need to be notified they have questions to complete.

Agreed. Not sure what the mechanism for this looks like yet. Could be a callback whenever a new question or form is created, or a job that goes and looks for incomplete profiles.

How does drag and drop work on smart device/touch screen? I have not worked in drag an drop in rails, nor for smart devices.

Desktop browsers can use drag and drop events which I believe are supported even in mobile browsers, although I'll have to research more into the mobile side to see how they work specifically. There are also touch events available in mobile browsers too. There do exist Rails plugins that can manage sort order but to be honest it's just an integer column + a controller action that accepts a new sort order from the frontend.

Is there a necessity for a ‘required’ form? My thinking is no. A form is something all users will need to answer (save for the user specific adopter/foster form which only each user type answers).

Agreed. I suppose if all questions in a form were optional then it would follow that the form is optional, but I don't think an explicit required field is needed on forms.

number of forms - so each response to a form question will be on the same model right? So is it feasible to have a separate form/submit for each form?

It can be done either way, the controller used to accept user answers can be structured to accept one form at a time or all applicable forms at once. There's just a single answers table that stores answers for all questions regardless of what form they are associated with.

On a related point, how can we handle validations? I am not quite understanding if we can have back end model validations or not.

We can hook whatever custom logic we want up to ActiveRecord validations. The Answer model will need a validator that looks at the input_type, required, and options of its associated Question and decide whether the value its about to store is valid or not.

This also relates to the separate form submissions, if we submit only part of a models attributes will validations for other attributes fail?

Creating a batch of Answer models will be done in a transaction, so if one Answer fails to validate regardless of whether or not its in a single form or group of forms, the whole batch will roll back and the user will be presented error messages to correct.

to @nsiwnf point above, some questions will have associated questions that e.g., anserwing Yes to 'do you have pets at home' would then show its related question of 'describe what and how many'.

Yes, if those sorts of questions can be generalized to an input type (e.g. "Yes/No + Explanation") then this data model should still work fine, a question's options could be extended to support custom labels and such.

[kasugaijin]

@uoodsq thank you for your responses - all makes sense! Please let me know how I/we can facilitate next steps on this work.

[mononoken]

Once Questions are added, a separate page exists that allows the staff user to drag/drop and reorder the Questions within the Form:

I was thinking, maybe we should consider adding buttons as well as an alternative way to adjust sorting. One possible version would be an up and down button on each question to move the questions. I like the intuitiveness of click-and-drag, but I think another more accessible option would be an improvement.

[kasugaijin]

Yes I think we can discuss in more detail how we think this should be implemented for MVP. My gut is we go with one solution for MVP, to keep things streamlined and moving, and that solution ideally works on desktop and mobile alike. In the case of drag and drop, I think we should get user feedback before building more than we need to. If what we build for drag and drop for MVP works well, win, if not, we can digest feedback and solve any problems.