What Makes Code Hard to Understand?

A (Quantitative) Cognitive Model of

What Makes Code Hard
reading & understanding

Quantifying Hard-to-Understand Code

• Complexity
  • Lines of Code (LOC) ← correlated w/ everything
  • Cyclomatic Complexity (CC) ← control flow
  • Halstead Volume/Effort (V/E) ← related to entropy
  • Percent comments (PC)
• Readability
  • # of identifiers ↓  (variable, function names)
  • Line length, indentation ↓
  • # of blank lines ↑
  • Software Readability Ease Score (SRES)
    • Like Flesch score (FRES)
    • Tokens = syllables, statements = words, units = sentences

The Psychology of Programming

• Cognitive Dimensions of Notation (Blackwell, 2001)
  • Consistency, hidden dependencies, etc.
• Rules of Discourse (Soloway, 1984)
  • Variable names should reflect function
  • Don't include code that won't be used
  • Don't do double duty with code in a non-obvious way
• Plans & Schemas (Pennington, 1987)
  • Textbase vs. situation/domain models
  • Effect of task
    • Read to recall/use/modify
  • Effect of experience
    • Reading strategies, schema use  (i for counter)

Understanding "Simple" Programs

• Understanding = predict output
• Simple = within metric thresholds

• Effects of Notation
  • Representation, spacing, ordering
    • Landy & Goldstone, 2007  (spacing & order of operations)
• Expectation Violations
  • Unused code, misleading schemas, priming
    • Detienne, 2001  (distorted recall)
• Interactions with Experience?

The Experiment

eyeCode

Format

Participants

• 162 participants (129 M / 30 F / 3 UR)
  • Bloomington - 29 ppl @ $10
  • Mechanical Turk - 130 ppl @ $0.75
  • Other - 3

Tobii TX300

Mechanical Turk

Variables

• Independent
  • Program version (2-3)
  • Python experience (years)
  • Programming experience (years)
• Dependent
  • Grade (0-10)
    • 10 = perfect, ≥ 7 = correct, 4-6 = CE
  • Reaction time (14 - 256 sec)
  • Common error (prob.)
  • Correct (prob.)

Results

counting

scope

whitespace

between

between (continued)

overload

What Makes Code Hard to Understand?

• Unconstrained Notation
  • Vertical spacing (grouping hints)
  • Horizontal spacing (facilitates calculation)
  • Mismatched ordering (efficiency)
• Expectation Violations
  • Recent calculations should follow from previous
  • Unused code should not be present
  • Single use of operator per block

Current Work

Eye-tracking &
Modeling

Eye-tracking: Areas of Interest

Syntax-based AOI Fixations

Raw

Corrected

Line-based AOI Fixation Times

Block-based AOI Fixations

Modeling

Thank you!

Questions?