Data Analytics / Chapter 07

What moves together?

A coefficient is a sentence about paired records. Change the records or target population, and the sentence changes.

Constructed examples. Under review; no public QR destination yet.

In this chapter
  1. One influential pair
  2. Choose a common case mix
  3. Look beneath a number
  4. Write a short practice report
  5. Code and captured output

One influential pair

The first four pairs stay fixed: (1,2), (2,1), (3,4), (4,3). Change the fifth response and compare distance with order.

Pearson
Spearman
Kendall-a (no ties)

Download executed Python lab

Choose a common case mix

A succeeds in 18/30 hard and 63/70 easy cases. B succeeds in 49/70 hard and 28/30 easy cases. Give both the same target mix.

A under target mix
B under target mix
Raw mixes: A 81%, B 77%

Choose a target weight from a real population, not a preferred winner.

Look beneath a number

Five points show positive linear association; an exact U shape has zero Pearson correlation.
A zero linear correlation can coexist with a fully determined curve. · Select the figure to enlarge.
B is ahead in both case groups while A is ahead overall under different group mixes.
The aggregate and the group comparisons answer different questions. · Select the figure to enlarge.
Two descending classroom patterns have a positive pooled association.
Keep group labels when the records have grouped structure. · Select the figure to enlarge.

Write a short practice report

  1. Define one pair and its unit.
  2. Inspect the scatterplot before choosing a coefficient.
  3. Record missing pairs, time window, groups and measurement resolution.
  4. Explain one pattern your chosen measure could miss.
  5. Keep the conclusion descriptive unless the design supports cause.
The reproducible notebook

Inspect the work.

Read the programs, run them in your own environment, and compare the results. The captured output records the chapter’s checked practical run; changing an input may change its answer.

Python association_lab.py
"""Executed, dependency-free numerical checks for Chapter 7. Invented data only."""
from math import sqrt, log2
from itertools import combinations
from statistics import mean

def pearson(x,y):
    ax=mean(x); ay=mean(y)
    num=sum((a-ax)*(b-ay) for a,b in zip(x,y))
    den=sqrt(sum((a-ax)**2 for a in x)*sum((b-ay)**2 for b in y))
    return num/den if den else None

def ranks(x):
    values=sorted(set(x)); out=[]
    for v in x:
        p=[i+1 for i,t in enumerate(sorted(x)) if t==v]
        out.append(mean(p))
    return out

def kendall_a(x,y):
    pairs=[(x[j]-x[i])*(y[j]-y[i]) for i,j in combinations(range(len(x)),2)]
    return (sum(z>0 for z in pairs)-sum(z<0 for z in pairs))/len(pairs)

x=[1,2,3,4,5]; y=[2,1,4,3,5]
assert abs(pearson(x,y)-.8)<1e-12
assert abs(pearson(ranks(x),ranks(y))-.8)<1e-12
assert abs(kendall_a(x,y)-.6)<1e-12
print('Five pairs: Pearson %.6f; Spearman %.6f; Kendall-a %.6f' % (pearson(x,y),pearson(ranks(x),ranks(y)),kendall_a(x,y)))
y2=[2,1,4,3,50]
assert abs(pearson(x,y2)-.7284)<.001
print('Extreme last y: Pearson %.6f; Spearman %.6f; Kendall-a %.6f' % (pearson(x,y2),pearson(ranks(x),ranks(y2)),kendall_a(x,y2)))
u=[-2,-1,0,1,2]; q=[t*t for t in u]
assert abs(pearson(u,q))<1e-12
print('Exact U-shape: Pearson %.6f' % pearson(u,q))
a_h,a_e=18/30,63/70;b_h,b_e=49/70,28/30
assert (18+63)/100==.81 and (49+28)/100==.77
assert abs(.5*a_h+.5*a_e-.75)<1e-12
assert abs(.5*b_h+.5*b_e-49/60)<1e-12
print('Channels: overall A %.2f B %.2f; common 50:50 A %.6f B %.6f' % (.81,.77,.5*a_h+.5*a_e,.5*b_h+.5*b_e))
phi=(12*7-8*3)/sqrt(20*10*15*15)
assert abs(phi-.282842712)<1e-8
print('Reminder pilot: RD %.2f RR %.2f OR %.2f phi %.6f' % (.6-.3,.6/.3,(12*7)/(8*3),phi))
partial=(.7-.8*.75)/sqrt((1-.8**2)*(1-.75**2))
assert abs(partial-.251976)<1e-5
print('Partial correlation: %.6f' % partial)
ca=[[1,5],[2,4],[3,3]];cb=[[8,12],[9,11],[10,10]];pool=ca+cb
assert pearson([a for a,b in ca],[b for a,b in ca])==-1
assert pearson([a for a,b in cb],[b for a,b in cb])==-1
assert pearson([a for a,b in pool],[b for a,b in pool])>0
print('Classroom within: -1, -1; pooled: %.6f' % pearson([a for a,b in pool],[b for a,b in pool]))
scale_x=[1,2,4,8,16];scale_y=[1,2,3,4,5]
assert abs(pearson([log2(t) for t in scale_x],scale_y)-1)<1e-12
print('Scale: raw Pearson %.6f; log2 Pearson 1.000000' % pearson(scale_x,scale_y))
print('All checks passed.')
Output Checked chapter run
Five pairs: Pearson 0.800000; Spearman 0.800000; Kendall-a 0.600000
Extreme last y: Pearson 0.728428; Spearman 0.800000; Kendall-a 0.600000
Exact U-shape: Pearson 0.000000
Channels: overall A 0.81 B 0.77; common 50:50 A 0.750000 B 0.816667
Reminder pilot: RD 0.30 RR 2.00 OR 3.50 phi 0.282843
Partial correlation: 0.251976
Classroom within: -1, -1; pooled: 0.896774
Scale: raw Pearson 0.933257; log2 Pearson 1.000000
All checks passed.