The Regional Crime Puzzle

Upon inspecting and understanding the unified dataset initially, we observed that its shape it has 119 rows and 24 columns, giving it a shape of (119, 24). From this, we noticed that the dataset is small-scale. Apart from its size, we also noticed that some columns need to be converted to a numerical value to be compatible with our study’s successive statistical tests. As such, we convert these columns (i.e., population_estimate_15_over, labor_force_estimate, poverty_rural, index_crime, index_crime_clear, nonindex_crime, nonindex_crime_clear, crime_sovled_total ) to a numerical value (either float or integer but in our case, we converted it to a float).

After data conversion to numerical values, some columns are irrelevant to the study’s aim. As such, said columns were dropped which cut the dataset’s shape to (119, 10). Apart from this, column(s) were renamed just for uniform’s sake (i.e., crime_sovled_total to total_crime_solved since we were getting annoyed with the spelling haha…)

From the unified dataset, we noted that some were not correctly up to scale, such as population_estimate_over, poverty_urban, and poverty_rural. In truth, their values were divided by 1000 from the dataset, which inclined us to multiply their values by 1000 to affix them to their true values.

Normally, this is done after outlier detection but since outliers may opt us to do this again… it was a tactical option to do this before outlier testing so that data cleaning (if needed) would likewise be only done once. Primarily, we engineered and added features such as total_poverty to further test our hypotheses and help in answering the research questions. More so, we ensured the inclusion of poverty and crime RATES to account for differences in regional populations. After all, larger populations may intuitively correspond to higher poverty, crime, and employment figures. Thus, we’re gonna be evaluating the effect of RAW COUNTS and RATES to crime on both scales! This then boomed our dataset’s shape from (119, 10) to (119, 18).

Proceeding with outlier testing, we ensured to avoid hastiness by merely picking some outlier testing and running our dataset on it. As such, we first ran normality testing using the Shapiro-Wilk test to check if our dataset was normally distributed. If it were, we ran a Z-score test. Otherwise, we ran an IQR test. From this, we noticed that the only "Likely Normal" feature in our dataset was total_poverty. As such, we ran every feature on IQR, whilst running only total_poverty on Z-score. From this, we got similar results, showing only one (1) critical outlier - an inconsistency to total_poverty_rate which amounted to 109%…. which is ABSURD! This suggested a data inconsistency in the original dataset. As such, this calls for a fix… which we carried out through data imputation!

To handle the absurd outlier mentioned above, we utilized mean imputation because we were concerned with both counts and rates anyways… This effectively resulted to an outlier-free dataset!

Since we would be using machine learning models in the latter part of the study (i.e., LinearRegression, PCA, Lasso, Ridge, etc.), standardizing the current dataset was a must. To do so, we carried out data standardization using RobustScaler() because our data was not normally distributed.

With everything cleaned and standardized, the datasets to be used for statistical testing and machine learning are now intact! Note that in this study, we will be using two (2) datasets:

• reinforced_df – unstandardized dataset for proper raw count vs rate comparisons
• standardized_df – standardized dataset via RobustScaler()for ML methods

Before explaining the answers to the research questions, do note that we tried four (4) (initially three (3) but due to uniform recommendations and subpar results... we opted to add another one) different approaches to determine and look for possible trends and correlations to properly answer the research questions. Direct Analysis, Tolerance of Gaps, Imputation and Time Series Forecasting, and Imputation + Principal Component Analysis (PCA) were the four (4) analyzed approaches. Each approach gave insights on the available data, and the last method, Imputation + PCA, served to be the most insightful for the answers. Learn more about the approaches done in our GitHub Repository.

Scatter plots were done to visualize the relationships of poverty and employment with crime. We compared urban poverty, rural poverty, and employment rate to total crime.

Using these data plots, there is no obvious pattern or trend among the collection of points. Urban-Crime seems to show a slowly rising trend with the more urban population having more crimes committed. The Rural-Crime graph shows a bit of random points, and the same goes for the Employment-Crime graph. A deeper analysis must be done in order to determine the underlying trends that is not so clear to the naked eye. A correlation heatmap is done to visualize and numericize the correlations.

Analyzing this correlation heatmap, it seems that urban poverty (poverty_urban) has the greatest correlation among the other variables with respect to the total crimes (total_crime) committed in a region. Then, employment rate (employment_rate) takes the second indicator; although, note that it shows a negative correlation. Finally, rural poverty (poverty_rural) lastly follows, showing a very close to zero (0) (i.e., absent) correlation to crime. There could be many reasons for this, such as a more "relaxed" life in rural areas or provinces and other cultural reasons. For now, we will keep it as is. This is not a worry as this observed relationship and reasoning will be further discussed in the successive findings. Nevertheless, this correlation heatmap directly answers research question number 1 (RQ1). Now, let us look at a more "regional" perspective and point of view.

Here, we would like to analyze the regional variations or in other words, compare the values in each region. We would mainly use the Imputation and Time Series Forecasting approach for this research question. First, we will show the time series plots and compare the regions.

From these graphs, we can see that the regions generally follow the same trends. Urban poverty rose from 2018 to 2021 possibly due to the pandemic, and some regions started to recover after the peak in 2021 and declined back through 2022 to 2025. Employment rate also took a huge drop in 2020 possibly due to the pandemic, but it rose back the following year in 2021 and continued to rise or remain stagnant until 2025. The total crime per region all have a declining trend from 2018 to 2025. There does not seem to be much of an outlier except for NCR, which was affirmed in the IQR test in the EDA, approximately having 140,000 total number of crimes during 2018.

Looking at the urban poverty of the regions and comparing their total number of crimes, we can see a slight correlation or pattern concerning the region. Specifically, observing the top five (5) regions in the urban poverty category namely, region 4A, region 3, region 7, region 10, and 12, all of them except region 10 and 12 are also found in the top five (5) number of total crimes committed, but with a slight difference in order. NCR has the most, followed by region 4A, region 7, region 3, and lastly, region 11. As explained earlier, there is a slight correlation amongst the number of urban poverty and total number of crime, and this holds true for the top five (5) regions. Moving on to employment rates, all regions do not differ much from each other. Taking the bottom five (5) regions in employment rate, we get NCR, region 4A, region 1, region 3, and region 5. Out of the five (5) regions, only three (3) of them are in the top five (5) of the total number of crimes, and only two (2) are also part of the top five (5) in urban poverty. Moving forward, looking at the opposite side and checking the bottom five (5) regions in the urban poverty category, we get regions CAR, 6, 2, 1, and 8. Now, for the employment rate (highest), we get regions, 2, 9, 11, 13, 12, and for total crime (lowest), we get BARMM, CAR, region 4B, 13, 9. From here, we can see that CAR (poverty), and region 9 and 13 (employment rate) are both found in the total crime. Region 4B is the 6th least number of people classified under urban poverty and region 13 ranked 6th. Likewise, region 9 ranked 7th least, which shows a bit of a relationship to the low crimes. However, BARMM, having the least number of crimes committed, actually ranks high in the urban poverty, ranking at 7th, and ranking 4th in rural poverty, and 6th lowest employment rate. Despite these counterintuitive placements, they rank the lowest number of crimes. This could be due to many factors such as religion and the 2014 peace deal ending the war in the region and the continuous seek to end conflicts.

Awesome! Now, let's look at the averages throughout the years in a bar graph to compare them easier.

Here, we can see the average crime rate seem to be higher in three (3) regions, namely NCR, Region 7 (Central Visayas), and Region 11 (Davao Region). There seems to be some relation on the location of a region with crime, which can be due to employment and poverty rates associated to the said locations. One thing to notice about this "Big 3" is that they are also relatively high on the urban poverty category in their respective regions, which can support the correlation found in RQ1. Stay tuned though as it will be shown in the Machine Learning Model section that the NCR region becomes an important feature for tree-based models!

This will be answered using our developed models. Feeling excited? Okay then... proceed to the Machine Learning Model section!

If both the independent feature (i.e., poverty and/or employment feature) and the target are approximately normally distributed, we carry out a Pearson correlation test. Otherwise, a Spearman correlation test is conducted. This approach allows us to quickly identify linear or monotonic relationships between each feature and the target variable. Similarly, group comparison tests are performed to determine whether the target variable differs significantly between subgroups. For this, we first split each feature into two groups based on quantiles (Low vs. High). Then, we apply Levene’s test to check for equality of variances. If variances are equal, we use an independent t-test (parametric); if not, we use a Mann–Whitney U test (non-parametric). This approach is used because the t-test assumes equal variances and normally distributed data, while the Mann–Whitney U test is robust to unequal variances and non-normal distributions. By dynamically selecting the appropriate test, we ensure that our conclusions about differences or associations are statistically valid regardless of the underlying distribution of the data.

Performing hypothesis tests to determine if poverty levels and/or employment levels have a significant effect on crime rates across Philippine regions, we took a branched granular approach. By granular, we mean that we did not only focus on raw numbers of poverty and employment to determine how they affect crime in the country as a whole. In actuality, we also considered rates of poverty and employment across regions because intuitively (without data in mind yet…), regions that are larger would seem to have a bigger scale and thus, a denser proportion in poverty, employment, and crime numbers. But does scaling regions fairly have a significant effect on crime rates or were raw counts enough to predict crime counts nationally?

After hypothesis testing, we can now confidently conclude the following:

• H₀,₁ (Urban Poverty): Poverty has no significant effect on crime → Rejected
• H₀,₁ (Rural Poverty): Poverty has no significant effect on crime → Failed to reject
• H₀,₂: Employment rate has no significant effect on crime → Rejected

Looping back to the final research question, the machine learning model will determine if improvements in lowering poverty and increasing employment will affect the number of crimes. To evaluate whether socioeconomic indicators can predict crime across Philippine regions, multiple machine learning models were tested using cross-validated R² scores. Two (2) separate targets were evaluated namely, Crime Counts and Crime Rates (per capita)

Linear Regression, Ridge Regression, Lasso Regression, Random Forest, and Gradient Boosting were used and validated using R². Linear Regression was chosen to look for any linear relationships between the variables. Ridge and Lasso Regression were chosen to reduce multicollinearity since poverty and employment may be correlated as seen from the heatmap. Random Forest and Gradient Boosting were also chosen to incorporate nonlinear relationships and regional differences.

To further ensure accurate and the best results, some simple tuning was done to the hyperparameters as seen in the picture above. The results are seen below. Again, for the full code and implementation of the models, please visit our Github repository.

This was also done to predict the crime rates using the same models, however, none of them showed to be effective or reliable.

From the results and validation using R², we can determine that Ridge and Lasso performed the best among the models with a score of 0.315 and 0.254 respectively. They performed the best due to Ridge reducing coefficient variance and Lasso setting irrelevant coefficients to zero (feature selection). This applies to this study since socioeconomic indicators such as poverty and employment share an overlapping variance, and regularization prevents overfitting while keeping the meaningful predictors. However, it is important to note that these ML outputs do not provide feature importance for Ridge and Lasso since key predictors came from hypothesis testing and not ML coefficients. This means that the main drivers were determined by the hypothesis testing and these models are only used for prediction. They work hand in hand to give meaning to the data.

These graphs show the performance and accuracy of the models. The more dots that are near or close to the dotted line means that it has a more accurate prediction.

Going back to the research question, the importance of a feature was retrieved from the tree-based models.

From here we can see that the population_estimate_15_over was the biggest factor or indicator for the tree models, and the NCR region came in as the 2nd most important factor. On the other hand, our main socioeconomic indicators, urban poverty and employment rate, are minimal contributors compared to the main ones. We simulated and tried to predict the effect of improving the socioeconomic factors by decreasing urban poverty by 10% and increasing employment rate by 5%. The predicted crime counts was modest with a ~32% variance (R² ≈ 0.32). This means that improvements in these factors reduce crime, but the remaining 68% are covered by other social or cultural factors. To summarize, the simulation confirms that both factors do affect the total number of crime, but it is limited as other unmeasured factors also contribute significantly to the total number of crime.