Understanding what is the next stage in DL (& ML) algorithm development: basic approach - Andrew NG on youtube
Terms: training, validation, test.
Split: training & validation 70%, test 30%
Procedure: cross fold training and validation, or further split 70% to training and validation.
BIAS - Situation 1 - doing much worse than human:
Training set error: 5% error (test on train)
Validation set error: 6% error (test on validation or CFV)
Conclusion: there is a BIAS between human expert and training set
Solution: 1. Train deeper or bigger\larger networks, 2. train longer, 3. May needs more data to get to the human expert level, Or 4. New model architecture.
VARIANCE - Situation 2 - validation set not close to training set error:
Training set error: 2% error
Validation set error: 6% error
Conclusion: there is a VARIANCE problem, i.e. OVERFITTING, between training and validation.
Solution: 1. Early stopping, 2. Regularization or 3. get more data, or 4. New model architecture.
Situation 3 - both:Human expert: 1% error
Training set error: 5% error
Validation set error: 10% error
Conclusion: both problems occur, i.e., BIAS as and VARIANCE.