Showing posts with label data. Show all posts
Showing posts with label data. Show all posts

Friday, May 2, 2025

The Missing Middle: Our Political Blind Spot

 I try to keep an open mind and critically examine information. When I hear arguments from the Right or Left, I find elements of truth in both. But reconciling the contradictions between them can be difficult. Then I was analyzing some data at work (unrelated) and came up with a way to piece both sides together. I call it "The Missing Middle."

In my experience, data can be overwhelming for many people, so to make this post accessible to more people, I'm going to keep it as simple and relevant as possible.

We are all familiar with the Bell Curve, which helps teachers assign grades in class. The problem with using it in education is that it identifies one side as good and the other side as bad. The general "Normal Distribution" curve used in science and medicine recognizes the middle as "normal" or acceptable, and the "tails" on either side as unacceptable. For example, blood pressure that's too high or too low is cause for concern.


I purposely created this example with "red" and "blue" to represent the Right and Left viewpoints, where I think most political discussions occur today. Just like blood pressure, it makes sense to raise alarms when either party drifts too far from the middle.

The problem is when either side treats every action of the opposing side as alarming. If a person is criticized for everything they do, they will stop listening to criticism.

The following are some current examples.

Immigration

Recently, the Right has made immigration an issue with threats to deport millions of immigrants. They criticize the previous administration for allowing millions to cross our borders and overwhelm our country. Over 260 alleged gang members were deported. Among those was a man who was mistakenly identified as a gang member. Critics argued that these deportees deserved "due process" - a fair hearing before removal. 

The Missing Middle

Of the 260 people, about 10-20 were convicted violent criminals (e.g. rape, murder). The government had been attempting to deport these people, but the cases were caught up in court. What's missing is a working immigration policy and more efficiency in providing "due process."

Government Excess

The Right has been attempting to rein in government excess, while the left points to critical programs that may be cut. I've heard the argument against D.O.G.E. that many government agencies already exist to do this work (Government Accountability Office (GAO), Office of Inspector General (OIG), Congressional Budget Office (CBO), etc.). 

The Missing Middle

While the current government auditing arms have already identified many areas for improvement, they aren't empowered to take action, and those empowered aren't required to listen to their counsel. It would be better if both sides figured out a way to leverage this missing middle.

Climate Change

The Left has been sounding the alarms for the consequences of climate change and trying to enact policies to reverse the increase in greenhouse gases. The Right mostly denies this and prioritizes human prosperity over planet health.

The Missing Middle

While humans have caused an increase in greenhouse gases, which has heated the planet, the planet has been warming up since the Little Ice Age (1300 to 1850 AD). So even if we could revert to conditions before the Industrial Age, the glaciers would still melt, and the seas would still rise. While some habitats are stressed by the changes, others thrive from increased CO2 and warmer temperatures.

More than 200 million people rely on the fresh water from the Indus River, which is glacier runoff. Scientists project a significant decrease in water by 2050. Without anthropogenic climate change, this devastation would only have been delayed.

The Right needs to acknowledge the coming changes and Left could be less alarmist. We should all work together to adapt to any unavoidable changes to our world and do our best to prevent what we can.

Conclusion

Focusing on the extremes of the normal distribution leads to polarization and unproductive discourse. It also distracts from much of the good that exists in the middle. We need to keep pointing out when either side strays too far, but we need to do it in the spirit of working together for the general good of the country. 







Sunday, November 3, 2024

Intellectual Vision: Four Truths and Eightfold Path

I’m trying to develop an idea like Buddhism’s Four Noble Truths and The Eightfold Path, which applies to human understanding of reality. 

The Four Truths

1) There is an absolute, objective reality 

2) Humans (as individuals or collectively) are incapable of knowing reality completely 

3) Humans need to understand reality as much as possible for our well-being and survival

4) Humans can best do this by following an Eightfold Path

An Eightfold Path

  1. Intellectual Humility – Recognize the limits of your understanding and be open to new insights, even if you consider yourself an expert. Recognize your own biases. Embrace the mysteries of the past and uncertainties of the future.
  2. Intellectual Curiosity – Cultivate a deep desire to seek knowledge and explore reality.
  3. Intellectual Honesty – Pursue and accept truth with honesty and integrity, even when it may be disadvantageous to yourself, your group, or your cause.
  4. Intellectual Access - Make collective understanding available to all people. Share critical knowledge openly and ensure access for everyone. Practice transparency.
  5. Intellectual Prudence
    1. Avoid sensationalism or oversimplification. 
    2. Critically evaluate information, verifying facts before accepting them or spreading them.
    3. Avoid jumping to conclusions or being swayed by misinformation. 
    4. Thoughtfully apply knowledge and consider how it affects others or society. 
    5. Practice responsible environmental stewardship by reducing waste, conserving resources, and supporting sustainable practices.
  6. Intellectual Tolerance - Accept diversity and differences, understanding that everyone has a unique life experience and perspective.
  7. Avoid Persuasion, Seek Understanding – Engage in dialogue to understand and establish a shared pool of understanding rather than to persuade, manipulate, or win an argument.
  8. Promote Empathy and Compassion - we are all human and make mistakes. We should not treat others harshly. It takes enormous energy to be informed on many subjects, and some truths are difficult to accept. “I’m striving to follow the eightfold path to truth. If I stray from the path, I appreciate earnest reminders from others. I recognize that not everyone shares my same conviction.

Example: Raising the minimum wage 

The Left and Right will make claims about what will happen if the minimum wage is raised. The Left claims it will give money to the people. The Right claims it will put companies out of business and may make it harder for teenagers or others to enter the job market, among many other unintended consequences. But no one knows the consequences for sure.  Let's look at this more openly.

If the minimum wage is raised, some businesses will comply, and others will not. Those who don’t comply may get away with it or face consequences.

Those that comply with it will pay for it out of profits or by raising prices. If they raise prices, they may gain or lose some of a particular customer's business.

If business decreases, the owners can cut costs by reducing their employees' hours, reducing the number of employees, or replacing employees with robots or other sources of automation. They could also find other cost-cutting measures like using cheaper materials, reducing energy use, or finding cheaper suppliers.

If employees leave (forced or voluntarily), they may find a better opportunity, the same, or worse. Some employees may stay enticed by the higher wages and miss out on other opportunities.

When replacing employees who leave, the employer may have an easier or more challenging time finding new employees. The employer may be more or less picky in their selection of future employees. If they are more picky, this could result in someone less qualified not getting the opportunity to get work experience.

If the minimum wage only affects a portion of businesses, it may be more or less challenging for other companies to find qualified workers (after the $20/hour minimum wage for fast food restaurants, my neighbor's company lost employees and ended up moving their manufacturing over the border).

Lower profits from any cause could end an ailing business, resulting in loss of employment for the people generously granted a higher wage. Some of these businesses may have ended regardless of the increased minimum wage, and there is no way of knowing at this point.

How each of the owners, employees, and customers responds is individual and based on that person's life experience and circumstances, and it can greatly differ by location (city, state, country).

The truth isn't necessarily black and white.

Resources

This post was inspired by Thomas Sowell's book Knowledge and Decisions. Specifically, the idea behind using the phrase "Intellectual vision" is from the book.

Intellectual Vision - a central set of premises from which particular positions can be deduced as corollaries . What makes them a coherent vision is the high degree of correlation among the particular conclusions reached among highly disparate subjects. An ideological vision is more than a belief in a principle. It is the belief that that that principle is crucial or overriding, so that other principles or even empirical facts must give way when in conflict with it.

You might call this post an "Intellectual Vision" to improve how ideas and beliefs can be formulated.

Monday, May 8, 2023

The Digital Data Bubble

Lately, I prefer a positive outlook on the future, yet I do have a couple of concerns. One of these concerns is data. The data we create each days grows exponentially, yet no one seems to discuss what this means long term. "Data storage is basically free!" is the usual claim. But at some point, I believe the Digital Data Bubble will have to burst. I can imagine a day when the monthly cost of maintaining data in the cloud will be a significant part of the household budget.

Thought Experiment #1 - A Messy Garage

Your garage is packed with stuff so you move to a house with a bigger garage. The new, bigger garage gets crowded so you move some of the stuff to a storage unit. The storage unit gets crowded, so you get a bigger storage unit. You keep getting more storage units until you find out about an empty warehouse. You move all of your stuff out of the garage and storage units into this one warehouse, and to your relief it barely takes up any space. You now begin storing all of your new belongings and stuff in the warehouse and to your surprise, it soon fills up. 

Is it time to get another warehouse? No. It's long past the time to get rid of stuff. Plus, you're frustrated because now you can never find anything and it seems that you have more junk than stuff of value.

Thought Experiment #2 - A Forest

The forest is thick with trees and plants. Countless animals and insects live there. A tree dies and falls over, cluttering the forest floor. Insects, fungus, bacteria swarm the tree and decompose it until the nutrients it held are released into the ground. There is a natural, ecological balance that prevents the forest from getting too dense.

Conclusion 

We need a strategy to purge our old useless data before it becomes too costly and too overwhelming to manage. 

Saturday, October 3, 2020

Looking at the Numbers Part 4: Follow-up on Previous Pandemic Insights

 I've been wanting to go back an evaluate my previous posts on the Pandemic:

Looking at the Numbers: COVID-19 New Cases

Looking at the Numbers Part 2: COVID-19 Cases in the U.S.

Looking at the Numbers Part 3: Insight into the COV-19 Pandemic using a simulation

TL;DR

  • Claim: the pandemic was following a log-normal distribution
    • While the log-normal is useful, the pandemic resembles a more complicated superposition of multiple log-normal distributions.
  • Claim: the future can be predicted using log-normal distribution and/or percentage of population
    • Partially true
      • I predicted total cases for CA, FL, TX would be much larger than expected at the time (actual numbers have exceeded my prediction).
      • Log-Normal cannot predict future outbreaks (Example: Italy, Russian, Japan and Spain)
  • My original claim that many populous countries were just starting to "blow" up and poorer countries will most likely do worse.  This is proven false in the case of Bangladesh and India with lower death rates than the U.S.
  • Claim: Re-opening will most likely result in a rise of cases
    • True

Log-Normal Distribution

Claim: A Log-Normal distribution appears to be a surprising good fit to the number of new cases in various countries

The claim seems to be mostly true if a country or state doesn't make major changes to their response to COVID.

For example, Brazil seems to be following a Log-Normal distribution for COVID-19 cases.  Below are the best fit cumulative and distributions.
Most other countries, however show a resurgence of cases.  Only the distribution function is shown.  For these cases, it appears that the distributions appear as a superposition of two or more log-normal curves. 





Predicting the Future

Claims: 
  • The log-normal is able to predict the future growth of the virus assuming no later waves
  • The trend shows that the maximum expected total cases will be about 2% of the population
  • Several populous states have a ways to go

The problem with this claim is that in all cases, there is a later wave.  Still, I believe that the log-normal can help provide an expectation of how a current outbreak in a locale will play out.  Some examples:

Second Outbreaks

On May 9, 2020, I predicted that Italy would have 246k cases.  It hit this number at the end of July (as predicted) however a month later, the cases started climbing again.  It retrospect, I remember looking at the regions of Italy that had been infected and I noticed there were many other populous areas with low infection rates.  So I am not surprised in the later rise, but I also had no way to predict when it would start nor how large it will be.
Other countries that have had secondary outbreaks (Japan, Russian, Spain).






Populous States

On May 24, 2020, I predicted that California, Texas and Florida would have significantly more cases based on the assumption that peak cases would be about 2% of the population.  This assumption of 2% ended up being too low for many states (Florida and Arizona are both over 3% of the population infected).

StatePopulationTotal Cases to DateEstimate from May 24Actual Oct. 3
CA39,144,81888,226694,670826,624
TX27,469,11452,268497,114766,559
FL20,271,27248,675356,750711,804


California's continually changing policies (partial shutdown, full shutdown, partial reopening, etc.) have resulted in a distribution that cannot be fit to a log-normal.


Populous Countries


On May 9, 2020, I noted that some of the most populous countries were just starting to see the pandemic: 
 India, Russia, Brazil, Mexico, Indonesia, Bangladesh, Pakistan, Nigeria.  Though I wrote in my post that  "It's too early to tell how this will play out worldwide", I alluded to my hypothesis that they would experience a more severe pandemic.  The table below does not support this hypothesis.  

CountryMay 9Oct 3Death Rate for those infected
United States1,283,9297,332,2852.8%
Brazil145,3284,880,5233.0%
India59,6626,473,5441.6%
Russia187,8591,194,6431.8%
Mexico31,522753,09010.4%
Bangladesh13,134366,3831.4%
Indonesia13,112295,4993.7%
Pakistan27,474313,9842.1%



Two cases stand out: India and Bangladesh.  India is quickly approaching the U.S. in number of cases but has a death rate nearly half of the U.S.  Bangladesh has an even lower death rate and the total number of cases are very small (0.2% of the population infected compared to the U.S. at 2.2%). And it appears that cases in Bangladesh are on a steady decline.

Re-opening will most likely result in a rise of cases

Here is my prediction using a scale-free model.  The total cases, time scale, relative size and time of peaks could not be accurately modeled.



Here is what happened in the U.S. after many states relaxed the rules around May or June.