Uncertainty Workbench
The Holistic Analyst and Brain Buddy test glass specimens and explore the results in Uncertainty Workbench.
Why a tool, and why now
In the first chapter of this series I argued that probability is a prosthesis. It is an instrument we invented to see something our senses do not report, and nobody is born knowing how to read it, any more than they are born knowing how to read an oscilloscope.
I meant that. But an argument about instruments is not an instrument. At some point you have to hand people something they can pick up.
That is what this piece is about. Over the past year I have been building a teaching tool. It started life as a Weibull analysis program for my own failure work. Somewhere along the way it turned into something else, and I renamed it to match. It is now called the Uncertainty Workbench.
I want to tell you what it does, what it showed me, and why I think its best feature is one that no real laboratory can offer you.
Ten coupons
Let me start with an example from an essay I wrote last December.
Ten samples of a material, pulled in tension. The results run from 438 to 471 megapascals, and the average is 455.4. Clean, tight, nicely behaved. Write down 455 MPa, put it in the report, move on.
Done, right? Wrong.
Here is what the Workbench says about those same ten numbers.
How much does it really vary? The Weibull fit gives a shape parameter, beta, of about 51 and a characteristic strength of about 460 MPa. A beta of 51 means a very consistent material. If it is true.
How well do we know it? With ten coupons, the honest 90 % range on beta runs from about 28 to about 69. That is the difference between a good material and an exceptional one, and ten coupons cannot tell you which you have.
What can you defend? The B10 strength, the stress at which one part in ten fails, is about 440 MPa. The value you can defend with 95 % confidence, the B-basis, is about 423. That is a 4 % penalty for having tested only ten.
What does the average hide? About 44 % of this population is weaker than the average. Not half, because strength distributions lean toward the weak side, but close enough that "455 MPa" is a promise half the parts do not keep.
And there is a fourth question the numbers cannot answer at all: how were they measured? The table gives no dimensions, no rig, and no calibration date. We will come back to that, because it turns out to matter more than I expected.
None of that arithmetic is hard. It is just arithmetic that nobody does when the back of the book has one number in it.
The experiment you can never run
Here is the problem with learning about uncertainty from real data.
In a real lab, you never know the truth. You test fifteen coupons, you get a beta, and you have no way to know whether that beta is close to the material's real value or a lucky draw. You cannot test the same material a thousand times to find out. Nobody has the budget, and the material would run out.
So you never see the thing you most need to see: how much your answer would change if you did the whole job again.
The Workbench's laboratories get around this with a simple trick. Every experiment starts from a material whose true properties are known. You type in beta and eta, or you pick a material from the built-in library. The software then tests that material the way you would, with however many coupons you choose, as many times as you like. Each answer can be put beside the truth.
That one change turns uncertainty from something you read about into something you watch happen.
The Uncertainty Lab, sampling experiment. Every grey line is one complete test program's answer; the black line is the truth no real test ever shows you.
There are two laboratories. The Uncertainty Lab has seven experiments: sampling, sample size, confidence intervals, the tail, choice of model, the size effect, and a comparison of 96 published Weibull fits for real materials. The Error Analysis Lab has five, all about measurement: the load-to-stress budget, rig scatter, bias and drift, rounding, and outliers.
Each experiment follows the same pattern. It asks a question. It asks you to predict before you run anything. Then it runs the experiment and explains the result using the numbers it just computed, not canned text. The prediction step is deliberate. You do not learn much from an answer you never had a stake in.
Let me walk you through a few of them.
Same material, different answer
Take a material with a true beta of 9 and a characteristic strength of 650 MPa. That is a reasonable stand-in for a decent technical glass. Test it with fifteen coupons. Then do it again, one thousand times.
1,000 test programs of 15 coupons from one known material. Ninety percent of the fitted betas fell between 7.0 and 14.0.
Ninety percent of the fitted betas land between about 7 and about 14. Same material. Same test. Same number of coupons. One program tells you the material is mediocre. Another tells you it is excellent. Both are honest results.
Notice too that the spread is lopsided. Small samples tend to overshoot beta. The typical answer is about 7 % too high. A small test makes a material look more consistent than it really is.
And here is the part that stopped me. In about 59 % of those thousand programs, the estimated B10 was higher than the true B10. Design to the estimate, and you are on the wrong side of the truth more often than not.
Your test result is one draw from this lottery. Only the material's scatter is real. The rest is luck, and the only thing that shrinks luck is more coupons.
What more coupons buy you
So how many is enough? Most test plans I review have never asked that question in numbers. The Workbench turns it into arithmetic.
For the same material, the B10 is about 506 MPa no matter how many coupons you test, because it is a property of the material. What changes is how much of it you can claim.
With 5 coupons, the B-basis value is about 306 MPa. That is 40 % below the B10.
With 15, about 427 MPa. A 16 % penalty.
With 30, about 456 MPa. A 10 % penalty.
With 100, about 482 MPa. A 5 % penalty.
Going from five coupons to thirty buys back about 150 megapascals of design stress without changing the material at all. That is a sentence I wish I could put in front of every program manager who has ever cut a test matrix to save a week.
The same tab includes a test planner. If you want to know beta to within plus or minus 20 %, you need about 45 coupons. Within 10 %, about 170. Those numbers surprise people. They should.
What "ninety percent confident" actually promises
A confidence interval is one of the most misunderstood ideas in engineering, and I include my younger self in that.
The Coverage experiment draws the intervals so you can count them. One hundred test programs of five coupons, true beta of 9. Each horizontal line is one program's 90 % interval. Some catch the truth and some miss it.
One hundred test programs of five coupons. Left: an exact method, 88 of 100 intervals caught the truth. Right: a common quick approximation, 70 of 100, with its misses on the dangerous side.
With an exact method, 88 of the 100 intervals caught the truth. Over two thousand programs, the rate is very close to 90 %. That is the promise, and it is a promise about the method, across many experiments. It is not a promise about your one interval. Once you have computed it, it either contains the truth or it does not.
On the right is a quick approximation that a lot of software uses by default. Only 70 of the 100 caught the truth. Worse, the misses are not evenly spread. With five coupons, the true beta fell below the interval about 22 % of the time, when it should have been 5 %. Missing low means claiming a more consistent material than you have.
So when software hands you a confidence interval, ask whether the method was built for your sample size.
Into the tail
Customers do not ask about one failure in ten. They ask about one in ten thousand, or one in a million.
A fit from fifteen coupons, read down to one part per million. Below the weakest coupon, every value is the model talking.
With fifteen coupons, the weakest one typically sits at about 4.5 % failed. Everything below that is not your data. It is the model, extrapolating.
The Tail experiment puts numbers on how far you are reaching. At one in ten, the estimate is about 506 MPa and the 95 % lower bound about 427. At one in ten thousand, about 234 against 128. At one part per million, the estimate is about 140 MPa, and the lower bound is about 57. Less than half.
The line on the plot goes on forever. Your data do not. A parts-per-million number from fifteen coupons is a statement about the model, and it needs checking against physics, against a second model, and against the field.
How was it measured?
The Uncertainty Lab is about the material. The Error Analysis Lab is about us, the people and machines doing the measuring. This is where building the tool taught me the most.
Measure what is squared. In a bend test, stress depends on the thickness squared, so every percent of error in thickness becomes two percent of error in stress. With calipers, thickness accounts for about three quarters of the measurement uncertainty in a typical four-point bend test. Switching to a micrometer on each specimen drops the total from about 0.9 % to about 0.4 %. Using the drawing's nominal thickness instead of measuring each specimen pushes it to about 2 %, and nobody would ever see that in the data. The cheapest reliability improvement in many labs is not a new machine. It is measuring the right dimension with the right tool.
Your rig widens your scatter. Random measurement error adds its own scatter to the material's, and you cannot separate the two afterward. For a true beta of 9, five percent random error per specimen makes the material look like a beta of about 8. At ten percent, about 6.5. Noise always makes the material look worse than it is. That is conservative, but you pay for it in allowable stress and in good parts rejected. Back to the ten coupons with a beta of about 51: for a material that tight, even a one percent rig accounts for about 14 % of the scatter you would see.
Bias hides. Drift can be found, sometimes. A constant bias, such as a mis-calibrated load cell or the wrong thickness typed in for every coupon, shifts every result by the same amount and leaves the scatter untouched. The Workbench's own drift screen flagged biased data in 4 % of runs. That is exactly the rate for perfectly clean data. A constant bias is invisible to statistics. Only calibration records and a reference specimen can rule it out.
Drift is different. A warming fixture, a wearing grip, a different technician after lunch: drift widens the scatter, and it can be seen if you plot strength against test order. But with fifteen tests, a 20 % drift across the run was caught only about a third of the time. With thirty tests, about three quarters of the time.
When a screen says "no trend found," it means it did not find one. It does not mean there is none. And none of it is possible unless someone recorded the order in which the specimens were tested, which is the single piece of information I most often find missing.
The outlier that was not
I want to be honest about this one, because the tool caught me.
The earlier version of my own program used a common style of outlier rule, the kind that assumes a symmetric bell curve. When I put it into the Error Analysis Lab and fed it good material, with a perfect test and no errors at all, it flagged at least one perfectly real specimen in about 16 % of fifteen-coupon programs. With thirty coupons, about 20 %. One test program in five would have had a real data point thrown away.
The reason is that strength data lean toward the weak side. With fifteen coupons of this material, the weakest one is expected to sit at about 71 % of the characteristic strength. A low point is what the weakest link predicts. It is not evidence of a bad test.
The current Workbench uses a screen calibrated for Weibull data. It asks the right question: given the other specimens, how unusual is the weakest one? It flags good data about 1 to 2 % of the time, by design. It also no longer recommends excluding anything. It reports how unusual the point is and leaves the decision with you.
The deeper lesson is not about the method. A flag is a reason to look. Look at the fracture surface. Look at the test log. Never delete a point because a statistic told you to. Delete it only when you can name the cause.
Your own data
The laboratories are for building intuition. The rest of the Workbench is for doing the work.
You can import your own results from CSV or Excel. Evaluate Data checks the test order for drift and says plainly what it cannot see. It runs the calibrated outlier screen, offers guidance on the model, and lets you pick the fit. From there you get a proper probability plot with confidence bounds, a choice of three bound methods, a comparison of fitting methods side by side, and a Failure-Rate Calculator that reports B-values, B-basis and A-basis allowables, and the small-sample penalty you are paying.
And there is a button I am particularly fond of: Explore my data in the lab. It takes your fitted result and makes it the "true" material for the experiments, clearly labeled as an assumption. Then you can ask the question that matters. If my material really is what my fifteen coupons say, how different could another fifteen have looked?
There is also a Learning Center with reference topics on Weibull analysis, uncertainty and measurement error. Its newest lessons start with questions rather than formulas: where uncertainty comes from, why the average is only part of the story, how to think about rare events, and how to record a decision so that you know when to revisit it. Every lesson connects to a live experiment.
A tool that teaches uncertainty must not mis-state it
I built the Workbench with AI partners. Claude did most of the development alongside me, and ChatGPT contributed a release candidate. Grok and Gemini each reviewed the code independently. I treat all of them the same way I treat a sharp colleague: as someone to argue with and be checked by, not someone to think for me.
The rule we worked by was simple. Every defect had to be reproduced by a failing test before it was fixed. No fix by assertion. The current version passes close to three hundred automated checks, and every number in this article came out of its engines.
That discipline paid for itself, and not always comfortably. Along the way the tests found:
Two coefficients in the confidence-bound formulas had been swapped. The intervals on beta were about 35 % too wide and those on eta about 16 % too narrow, on every plot. Simulation caught it.
A "bootstrap" option that quietly returned the ordinary bounds while still calling itself bootstrap.
A drift check that had never worked. It ran on data that had already been sorted, so every dataset looked like a perfect upward trend.
The outlier rule I described above.
An error in my own audiobook. I had said that the B10 is about 45 % of the characteristic strength at a beta of 5, and about 80 % at a beta of 20. The correct values are 63.8 % and 89.4 %. A listener cannot check arithmetic in their head while driving, which is exactly why it has to be right. A regression test now guards those numbers.
I tell you this not as a confession but as the point of the whole exercise. Everything in this series is about the gap between what we believe and what the evidence supports. A tool that teaches that has to hold itself to the same standard. The errors were real, they were mine, and the only reason I know about them is that I built something designed to check.
What it is not
I want to be clear about the limits, because a tool that oversells itself is teaching the wrong lesson.
It is a teaching and analysis tool, not a certification. It does not replace a qualified statistician on a program where lives depend on the answer.
Its strength is the Weibull distribution for complete strength data. It does not yet handle censored data, meaning parts that survived the test or were removed early. That is on the list.
A fit is not physics. A beta near 2 does not prove a mechanism, and a characteristic strength is not a guaranteed minimum. The tool says so in its lessons, and I am saying it here.
It runs on Windows. It has been tested thoroughly on my own machines, but not yet across every display setup and screen reader, and it is still a release candidate.
Why I built it
I have reviewed a great many datasets in forty-six years. The ones that went wrong were rarely wrong because someone could not do the mathematics. They went wrong because nobody had ever seen what fifteen coupons can and cannot tell you, so nobody thought to ask.
You cannot see that from a textbook. You can see it by running the same test a thousand times in an afternoon and watching the answers scatter around a truth you happen to know. Something changes in how you look at the next real result you are handed. I have watched it happen in other people, and it happened in me.
That is the whole idea. Not a better calculator. A place to build the reflex of asking how sure, before you need it.
The one-sentence version
Every strength value you are handed is one draw from a lottery you cannot see; the Uncertainty Workbench lets you see the lottery, by testing a material whose truth you know as many times as you like, so that you learn what your data can and cannot tell you before it matters.
The Uncertainty Workbench is a teaching tool by Joseph P. McFadden Sr., built with Claude and ChatGPT and independently reviewed by Grok and Gemini. Every number in this piece was computed with its engines. I am a materials engineer and failure analyst, not a statistician; the judgments here have been tested against forty-six years of things that broke. If you have a dataset that does not behave the way its report says it should, I would like to hear about it.
