Background
The mission of the California Department of Real Estate (DRE) is to safeguard and promote the public interests in real estate matters, which it accomplishes through licensure, regulation, education and enforcement. To achieve this, the DRE must first establish a standard of knowledge for licensed real estate agents and then measure this knowledge with an exam.
California has the largest real estate license program in the United States. To support the program, the DRE team develops and maintains a huge volume of exam items. This involves a rigorous process of exam validation every 5 to 7 years, to ensure items are legally defensible and appropriately testing for entry level into the profession.
As an existing provider of proctoring services to the DRE, and a full-service provider, PSI was well positioned to respond to the scope of work for exam development as an additional service.
In early 2021, DRE contracted with PSI to provide item review, gap analysis and item writing for Broker and Salesperson licensure exams, as well as the Professional Responsibility exam.
The Challenge
Huge volume and tight turnaround
DRE’s existing item bank contained thousands of items, which the PSI team needed to review in less than 18 months to hit the project deadline. A consensus decision was required from a DRE Exam Review Committee (ERC) of Subject Matter Experts (SMEs) whether each item needed to be retained, edited or deleted. Following a decision, item editing and writing needed to be managed and completed with input from SMEs and the DRE within the project timeframe.
Switch to virtual
As a result of the pandemic, the PSI and DRE teams needed to quickly pivot. Plans changed from in-person meetings to completely virtual ERC meetings almost overnight. This was a big shift from the weeklong in-person workshops DRE traditionally held pre-pandemic.To maintain security in face-to-face meetings, SMEs were effectively locked in a room with no access to cell phones. They were not allowed to remove their notes or any paper from the meeting. A difficult environment to replicate virtually.
SME engagement
With the move to virtual meetings, and with security still of utmost importance, the processes involved with SME recruitment and communication were crucial. High standards of SME participation and contribution needed to be maintained during and in-between meetings. Along with a highly controlled and focused environment through every virtual meeting.With a move to virtual, previous incentives of travel and accommodation no longer existed. Different methods were needed to recruit and engage SMEs. Establishing clear rules and expectations from SMEs in these unprecedented circumstances also was vital.
Establishing trust and consistent communication was key. Once the DRE team understood our process and how we would manage virtually, everyone was dedicated and on-board. They could see the flexibility and the greater outcomes. They were an amazing group to work with.”
Chanda Turnbull
Managing Test Development Specialist, PSI
PSI took the security of our exam content very seriously. I also appreciated their flexibility and communication. The PSI team ensured the whole process was controlled and straightforward for us.
Jeff Oboyski
Assistant Commissioner, Licensing California Department of Real Estate
Solution
Establishing SME relationships
Building a rapport with SMEs was important. Everyone was experiencing new and different challenges because of the pandemic and we all worked together to adjust. As there would be no travel costs, the financial honorarium per 4-hour meeting was increased for SMEs. As well as continuing education credits for participation.
It was also clear that the SMEs valued the opportunity to contribute to the development of their profession during a difficult time. And many participants appreciated the reduction in travel time required for meetings.
All ERC meetings were organized and facilitated by PSI Test Development Specialists, with at least one DRE representative also present. Virtual meetings were a big shift, but the change in format worked well. Focus was maintained by keeping meetings to a maximum of four hours and avoiding meetings on consecutive days.
Client collaboration and trust
PSI worked hard to gain the trust of the DRE team with our expertise and regular communication. This included monthly update calls and additional meetings when needed.
Effective processes and facilitation
The focus of ERC meetings was to review the existing item bank and revise items where needed. Each item was reviewed and approved using a group consensus decision.
The ERC reviewed all items for technical accuracy and to ensure items were practical, important and job-related.
Specifically, ERC members reviewed each item and verified the keyed option was correct and that the distractors (un-keyed alternatives) were incorrect. ERC members also modified the wording as necessary to ensure the item was easy to understand and reflective of industry practices in the state.
The panel deleted an item from the item bank if it determined it was for any reason not appropriate or relevant to a minimally competent licensee. SMEs were also asked to confirm if each item was appropriate for Broker only, Sales only, or both Broker and Sales.
We really appreciate that through the most challenging of times, PSI maintained high levels of security and worked closely with us to securely run a high volume of virtual SME meetings over just a few months. At the same time as keeping participant SMEs committed to seeing the process through.
Jeff Oboyski
Assistant Commissioner, Licensing, California Department of Real Estate
We weren’t sure how the change in meeting format was going to look. But, we ended up getting more done in shorter online meetings than in the longer face-to-face workshops. Participants weren’t checking out mentally after 6 hours due to being tired and fatigued. So, the quality of outcomes was better.
Eileen Brewster
Licensing Manager, California Department of Real Estate
SME training
Prior to reviewing items, SMEs participated in a virtual training session. The PSI team discussed the overall goals of the ERC meetings, steps in the review process and characteristics of items.
SMEs who agreed to accept item writing assignments also received item writing training in a 2- hour webinar. This session covered standard item terminology and item-writing rules, including specific style requirements of the DRE. Each panel member received a copy of PSI’s item writing training presentation slides.
Gap analysis, new and rework items
Following the initial review of all items, PSI completed a comprehensive gap analysis to identify areas where new items were needed. The number of items on a range of existing topics was increased and/or updated to reflect both new laws and emerging regulations. These included topics such as responsible broker supervision, marketing & advertising, property management, fair housing, adaptive reuse, and accessory dwelling units. The new and expanded items were identified by DRE staff and validated by SMEs.
PSI also considered items that were placed in rework status and strategically assigned a portion of these to individual SMEs for revision. Twelve SMEs participated and each was assigned a unique item writing or item rework assignment. Eight sessions were held with SMEs to review and finalize new and rework items.
Emerging innovations
A good example of innovation in the face of the pandemic was enhancements to the process for data transfer. Data previously delivered physically on password protected CDs is now shared through secure online data transfer. It’s fast, easy and can be done on the fly – in quantities that would have required multiple CDs.
Another area where the DRE embraced innovation and will permanently adopt change was the recent implementation of the Online Exam License Application (OELA) system. Applicants are now able to apply for an exam or both the exam and license online. In addition to submitting an application, OELA allows users to upload any necessary documents, pay the required fees, and check the status of their application all online.
We’ve learned that the virtual world doesn’t have to be as intimidating as we once thought – as long as you have the proper controls and systems in place.
Jeff Oboyski
Assistant Commissioner, Licensing California Department of Real Estate
Gap analysis is the process of identifying the number of items assigned to each topic or subtopic area of the content outline. We then make determinations if additional items should be written to fill in the gaps or areas which have less items.
Andrea Dominiack
Director Test Development, PSI
Output
This project was a full team effort, which involved multiple PSI facilitators and up to three virtual meetings per week. We reviewed thousands of items and developed new item content in less than a year.
Jeff Oboyski
Assistant Commissioner, Licensing California Department of Real Estate
Future Plans
Having completed the process of item review, item writing, making necessary changes and finalizing the item bank, the final step is creating the new exam forms. It will then be time to create a scope of work for the next item validation process, incorporating practices and strategies from the last validation to become part of the long-term solution.
DRE will be adopting a hybrid approach to SME meetings going forward. In the right format, virtual meetings have proven to be extremely effective. But there are still clear benefits to being physically present. The way forward for DRE involves taking the best from both the virtual and physical worlds.
"I would underscore that PSI was consistently committed to meeting our deadlines through the many challenges of a pandemic. It’s impressive how dedicated they were to see the project through and get the job done during a period of unforeseen roadblocks."
Jeff Oboyski
Assistant Commissioner, Licensing, California Department of Real Estate
