By the FoxyAI Team

We were asked to write an essay that “validated our scientific foundation” this week as part of a company presentation.

Huh? – It was a different exercise than an engineering code-check. We had some fun with it and this is where we landed.

FoxyAI Scientific Validation

1. INTRODUCTION

FoxyAI has developed a machine learning model library encompassing tens of real estate specific computer vision models. These models address various use cases with different model types, including object detection, instance segmentation, image classification, image regression, and image-to-image generation. Our research team trains these models using high-quality, diverse datasets and evaluates them using standard metrics such as precision, recall, accuracy, mean absolute error (MAE), and mean squared error (MSE). FoxyAI continuously develops, audits, and updates models as an integral part of its research operations. This document outlines FoxyAI’s scientific process and highlights results from our most recent models, validating our commitment to data-driven methodologies.

2. METHODOLOGY

FoxyAI adheres to a rigorous, iterative process when developing our machine learning models. This approach is designed to ensure that each model meets the highest accuracy, efficiency, and reliability standards while adapting to the evolving needs of the real estate sector. The following sections detail the key stages of our process:

2.1 Dataset Creation

The foundation of any machine learning model is the quality and comprehensiveness of the dataset. At FoxyAI, we prioritize the creation of diverse and representative datasets that capture the wide range of scenarios encountered in real estate applications. These datasets are updated as data drift occurs to ensure our models always provide up-to-date results. Our data curation process involves:

1. Data Collection: We gather data from multiple sources, including real estate listings, partner datasets, customer datasets, and publicly available datasets. This data is carefully selected to ensure it covers various property types, geographical regions, and market conditions.
2. Data Preprocessing: The collected data undergoes preprocessing to enhance its quality. This includes cleaning the data to remove noise, normalizing the images for consistent input, removing duplicates, and augmenting the data to artificially increase the dataset’s size and variability, improving model generalization. Depending on the task, we also exclude images containing indicators of a resident’s religion, race, disability, sexual orientation, or other personal attributes to minimize the risk of introducing bias into our models.
3. Annotation: Data is meticulously annotated by domain experts from leading data labeling firms. FoxyAI researchers and subject matter experts are actively engaged in the review and quality assurance process, maintaining direct oversight to guarantee the integrity of our datasets. Data review is never outsourced.

2.2 Model Training

With the dataset prepared, FoxyAI employs state-of-the-art training techniques to build robust machine learning models. Our training process is characterized by the following:

1. Model Selection: We continuously review the latest research to stay at the forefront of advancements in model architectures and pre-trained networks. Cutting-edge architectures like Transformers and Convolutional Neural Networks (CNNs) drive our focus on maximizing accuracy and efficiency.
2. Hyperparameter Tuning: At FoxyAI, we iteratively train models with a wide range of hyperparameters to achieve the best performance tailored to our specific tasks and datasets. We leverage advanced optimization techniques, including grid search, random search, and Bayesian optimization, to efficiently explore the hyperparameter space. This approach allows us to strike an ideal balance between accuracy and computational efficiency, ensuring our models are both effective and resource-efficient.
3. Model Validation: Validation is conducted using a distinct validation dataset that is separate from the training data. This validation dataset allows for continuous monitoring of the model’s performance throughout training, enabling the adjustment of hyperparameters and other model configurations in response to performance metrics. By evaluating the model on this validation dataset, we mitigate the risk of overfitting and ensure the model can effectively generalize to novel, real-world data.

2.3 Model Evaluation
FoxyAI’s models are evaluated with rigorous standards, utilizing a comprehensive set of performance metrics to assess each model’s effectiveness. After the initial validation phase, the final evaluation is performed on an isolated test set. This test set, which has been withheld during training and validation, is the definitive benchmark for the model’s generalizability and robustness. We utilize the following task-dependent metrics:

1. Precision, Recall & F1: Classification, object detection, and instance segmentation
2. Accuracy: Classification
3. MAE and MSE: Regression
4. Qualitative Evaluation: Image-to-image generative tasks

A thorough analysis of the test set results is performed to gain deeper insights into the model’s decision-making process. By identifying and understanding instances of false positives, false negatives, and other errors, we iteratively refine our dataset and training processes to minimize undesirable outcomes and enhance model performance. This evaluation process is conducted regularly on new test sets to detect and account for data drift, ensuring our models remain robust and effective over time.

3. MODEL RESULTS AND DISCUSSION

This section will showcase the research process and outcomes of a recently released model.

3.1 Recently Released Models

1. FoxyAI Lock Installation Detection

The FoxyAI Lock Installation Detection model is an object detection system designed to identify the circular hole in a door after the lock or doorknob has been removed, signaling the lock is being replaced. This task is particularly challenging due to the model’s susceptibility to mistakenly identifying any black circle, which can appear in various real estate images such as drain holes, vent openings, light sockets, etc. To mitigate false positives and enhance precision, the training dataset was fortified with a wide variety of potential false-positive scenarios.

After many dataset iterations, the final model was trained on 900 images for 353 epochs with a batch size of 16. We trained using techniques such as image augmentation, learning rate warmup, learning rate decay, dropout, quantization, batch normalization, and the Adam optimizer. This model has an inference speed of 50ms per image on a T4 GPU (hardware).

2. Results

The model achieved an F1 score of 0.978 on a test set of 250 images, encompassing a diverse array of challenging cases, including many at high risk for false positives.

F1 is a way to see both precision and recall as one metric. A perfect score is 1.000.

FoxyAI Lock Installation Detection Test Results

4. CONCLUSION

FoxyAI’s scientific foundation is built upon a rigorous, systematic approach to machine learning model development tailored to the unique challenges of the real estate sector. Through meticulous data curation, advanced training methodologies, and comprehensive model evaluation, we have developed a suite of models that demonstrate high performance and reliability. To ensure our models remain current and effective, we utilize a scheduled rotating model audit and review program incorporating the latest datasets, AI/ML best practices, and performance reviews. The results and processes detailed above validate FoxyAI’s scientific and technological foundation.

And, there you have it!