What do practitioners need to know about regression? - Statistical Modeling, Causal Inference, and Social Science:
"More specifically, here are some tips:
- The difference between 'significant' and 'non-significant' is not itself statistically significant.
- Don't just analyze your variables straight out of the box. You can break continuous variables into categories (for example, instead of age and age-squared, you can use indicators for 19-29, 30-44, 45-64, 65+), and, from the other direction, you can average several related variables to create a combined score.
- You can typically treat a discrete outcome (for example, responses on a 1-5 scale) as numeric. Don't worry about ordered logit/probit/etc,, just run your regression already.
- Take the two most important input variables in your regression and throw in their interaction."