How good are your appraisers, really?

Some years back, I wrote a paper on the state of the appraisal industry for a prior employer of mine who may not always succeed but nonetheless is too big to fail: We asked almost every question you could think of, and our answers did not paint the industry in the best light. Here are just a few of the questions we investigated:

1) Are appraisers influenced by the contract price?

2) Is there evidence that appraisers are aware of the loan terms borrowers are seeking, even though Dodd-Frank clearly forbids this?

3) How does an appraiser’s skill deteriorate as he moves away from his “center of work” (i.e. the area in which he works the most often)?

4) Do appraisers lie about property quality and condition?

5) How often do appraisers contradict themselves when describing the features of a comparable? And, more importantly, are these errors random, or do they tend towards making the comparable, well, more comparable?

6) Do appraisal management companies (AMCs) improve appraisal quality? How do various AMCs stack up against one another?

7) How do appraisers perform relative to their electronic competition, the despised computer (also known as an AVM, or asset valuation model)?

8) How much better are the best appraisers than the worst?

My research suggested that while many AMCs do add value (though there are those who struggle with QC) the industry, nonetheless, needs ways of tracking appraiser performance more closely. This is the first post in a series tackling ways your company can do just that. Let’s look at the last two questions above. We will save the others for future posts.

During my research, I used a well-known AVM to benchmark appraiser performance. I took the values from REO appraisals—i.e. appraisals used to help price foreclosed homes for sale— and marked them to market against the subsequent sale. I then repeated this exact procedure using the value the AVM produced when it was evaluating this appraisal. (The AVM was valuing the property as of the appraisal’s effective date, and I used the same zip code level HPI to mark-to-market it). I was careful to limit myself to REO sales where no improvements or repairs were made to the property after the appraisal was performed and where the subsequent sale occurred no more than three months later. While I cannot share my exact findings, it was clear to me that appraisers as a whole were not exactly running laps around the AVM despite the role the appraisal played in setting the REO’s listing price, something that clearly works in the appraiser’s favor.  

That said, there were major differences in appraiser performance. Some were notably better than the AVM, and others were quite a bit worse. These differences persisted even after I controlled for geography and assignment complexity. [At Kukun, we have an assignment complexity score too—something quite a bit better than my initial research produced, employing superior methods that were not available back then]. This suggests that there are meaningful differences in skill among appraisers. While delivering loans to the GSEs might be nothing more than a perfunctory exercise in box checking, banks that plan to hold loans in portfolio care how well the appraisers they are hiring perform. Accuracy matters.

How significant was the dispersion I found between the best appraisers and the worst? While I don’t believe I am at liberty to say exactly, they were profound. The best appraisers perform better, in fact, than the typical realtor, who—truth be told—tends to produce more accurate values than the median appraiser.  (I have determined this by comparing initial listing price to final sale price. That said, a more thorough analysis would involve performing a similar exercise to the one described above using BPOs, so I know many will disagree with this assertion. We all are aware that realtors play games with listing prices, so it is fair to say they don’t necessarily represent the realtor’s opinion of value. However, this fact tends to work against the appraiser. Of course, we also know realtors know the properties they are selling far better than those on which they perform a BPO, a fact that works in the other direction and favours the appraiser.) None of the above is mentioned to disparage appraisers, but the fact that performance varies among employees is something every business must acknowledge.

Do you want to know how well your appraisers are performing? Do you want to anticipate GSE feedback ahead of time and avoid painful back and forth with your clients? Our products can help you do that. They were built by the same people who helped develop Fannie Mae’s Collateral Underwriter™—we can anticipate GSE feedback and cut down your turn times.  We can track the performance of your appraisers using a system that doesn’t cut corners or get tired—and that even accounts for the complexity of the assignments they are given.

Consider working with our partners at TryJaro.com and purchasing our combined suite of appraisal feedback and analytics. Our AVM is only the beginning.  

Franklin Carroll

Franklin Carroll is Kukun's VP of Modelling and Analytics; After finishing his studies at the University of Chicago, he worked at Fannie Mae where he helped build Collateral Underwriter, roll out the Value Verify initiative, and specialized in collateral related and credit related policy.

Published August 24, 2023