> CEBq @bjbjt+t+ RAA<]44444@4````````$ V`````````r``,,`*`T#4411/Dec/2011
Hi Dave,
Back on 22nd of last month you sent me 3 documents that Norbert has proposed for inclusion in an update of AIAA R-091-2003. Im sorry to be a bit slow in getting back to you with comment. The 3 documents were:
1. ChapterForAIAADocumentV3.docx
2. Ulbrich-MathTermSelectionChapter.docx
3. Ulbrich-PercentContributionChapter.docx
and I'll take them in order.
1. SUBJECT: New Chapter for AIAA Recommended Practice
In the past Norbert and I have exchanged quite a number of E-mails discussing this subject matter, but as Im not a statistics expert I dont feel qualified to offer any critique of the statistical processes being described. I can say that, when I first became aware, (in late 2007), that Norbert was working on ways to determine objectively which math model terms were NOT supported by any given loading design, I was really interested to learn more about it. That was a problem that had bugged me for years, and one for which I had never found a satisfactory solution! But I fairly quickly became quite wary of the process Norbert was implementing, as it seemed to me it was being done with an insufficient appreciation on his part of the 'real experimental world', and how balances were designed and used in experimental testing.
While completely supporting the intent of Norbert's statistical approach, I find myself in disagreement with some of his premises. From the start one point of disagreement has been his interpretation that the AIAA recommended model consists of 10 'groups' of terms (not including the intercept term), as presented in Eq. 3.1.3. It was I who introduced the extension of Eq. 3.1.2 to Eq. 3.1.3, with the sole purpose of enabling global regression solutions for math models that catered to load asymmetry, as that was something the WG members felt was necessary with multi-piece balances. I maintain that Eq. 3.1.2 remains the basic governing equation, and that equation contains only 4 'groups' of terms, (indeed only 3 if one considers the Fj2 term to be included within the Fj.Fk grouping). What I dislike about Norberts interpretation is that it has resulted in his 'optimization' process considering all terms to be unrelated, and thus judged for inclusion in or exclusion from the math model only on the statistics for each term singly. I consider the 'signed' and 'absolute value' terms in each of the 4 (or 3) 'groups' to be interdependent, and have suggested to Norbert that there should be an additional constraint imposed on the term selection procedure, but I was never able to persuade him of this. In case you might be interested, I've attached a copy of a document and spreadsheet I put together a couple of years back when we were debating this issue, ("The background history of the 6x96 balance math model.pdf" and "Coefficient transformations.xls").
Over an approximately 2-year period a couple of years back, I had occasion (while supporting Dennis at FMS) to analyse data from calibrations of several balances that Norbert was also analysing. In one of those cases (Ames Mk.2A balance) I explored the idea of imposing the additional constraints noted in the above PDF, and documented my findings in "Statistics summary plots for 9 matrices.pdf" which is also attached. I explored 6 additional math models, 5 of which contained even fewer non-zero terms than Norbert's 'optimum' matrix, and sometimes substantially fewer. Yet for the 'check loads' computed from manual load and 6-component test load calibrations, the residual errors they produced were generally as good as, or even smaller than, the errors produced by Norbert's 'optimum' matrix. Of course, for the 'parent' calibration dataset the magnitude of the error standard deviations tends to correlate with the number of terms in the math model, so the full 6x96 model gave the lowest values and the model with the fewest non-zero terms the highest. While just based on a single balance and calibration, I thought the result was interesting enough to warrant further investigation, but Norbert was not persuaded.
Subsequently, in analyses of data from 2 other balance calibrations performed by FMS for Ames, (FMS MC60-D and a 3-component Canard balance design), I demonstrated that the so-called 'optimum' matrix determined using Norbert's BALCAL program, quite often did not produce the lowest residual errors in computations of check loads. Assuming some degree of independence of those check loads from the calibration loads, those results should be at least somewhat indicative of the predictive abilities of the models being tested. In these two cases I evaluated the math models from scratch; the selection of terms to be included (or excluded) was made using my own non-rigorous, and somewhat subjective, assessment process, rather than by just imposing an additional constraint to modify Norbert's 'optimum' model as determined by BALCAL. (Interestingly, the subject of Norbert's 3rd proposal, 'Percent Contribution', is the basis of how I have assessed math models for many years!) In the attachment "Canard_Calibration_Discussion.pdf", I have summarized part of a chain of E-mails on the Canard balance issue, exchanged between Ray, Norbert, Dennis, the client for the Ames test and myself. Also included are some results for both the Canard and MC60-D balances, comparing several alternative math models to the "BALCAL 'optimum'" model. These discussions had been prompted by the fact that the BALCAL optimization process had not worked very well in the case of the Canard balance, where it produced an optimum model that excluded 2 very significant linear interaction terms, and consequently the matrix performed very poorly indeed. This was definitely an instance of the baby being thrown out with the bath water! And also a demonstration that a good understanding of practical applications of balance technology trumps statistical expertise when it comes to practical situations!
All of the foregoing doesn't do much to address the specific content of the #1 proposal, and in any event I really don't feel qualified to critique what is essentially a discussion of statistical theory and processes. As Ive already said, I fully support the objective of providing a means to match an appropriate math model to the load design in an objective way. However, I'm more than a little afraid of what might happen should the process Norbert is championing (BALCAL) be used in 'black box' mode by test engineers. Unless those test engineers have a good understanding of balance technology, Im afraid theres a risk that some may be seduced by the idea that a computer program applying the statistical processes described, will give all the correct answers! As the Canard balance result demonstrates, that is not always so, and even the programs author can get burned if too much reliance is placed on the program and not enough on knowledge of basic balance technology!
About the only part of this proposal that I'll pass specific comment on is in section 3.1.2.2 Math Term Selection where it says "... Initially, the analyst has to decide if balance calibration data is to be processed using the iterative or the non-iterative method. This decision determines if balance loads or strain-gage outputs are the independent variables of the global regression analysis problem. ...". Apart from the fact that it should read data are rather than data is, this statement in the new proposal seems to me to be quite at odds with the original document's recommendation that the iterative formulation was the preferred model! Section 3.1.1.1. discusses the two math models, and explains quite fully the rationale behind the choice of the iterative model, where the balance loads are the independent variables.
2. SUBJECT: Math Term Combinations (fourth revision)
Personally I dont think there is much to be gained by including any of this material since it is really only repeating information that has already been covered earlier, in the preceding 4 paragraphs in section 3.1.1! Neither do I favour the 'spelling out' of options 1 to 9, nor the classification of options based on single-piece or multi-piece balance designs. It's the actual behaviour of the subject balance that should guide the choice of terms to be included, and I think that choice should be made by examining each situation on its merits rather than by checking one of several pre-defined "options". Norbert seems to be fond of compartmentalizing everything, but I favour having the balance calibrators and balance users think for themselves and understand the available choices! And I might add that the desired test accurary should also be considered when defining any balance calibration load design, (which in turn will set limits on what math models are possible), since it may not always be necessary or cost-effective to perform an elaborate calibration.
However, should the group members be in favour of the listed options approach, I would still question the logic of specifying both 2 & 3 and 4 & 5 as separate options -- the Fj2 terms are just a subset of the Fj.Fk terms, thus rendering options 3 and 5 redundant. In my view it would have been more logical to specify the #2 and #4 options as including the Fj2 terms but excluding the 2-load products (Fj.Fk), thus making it possible to determine those models from calibrations made with just single-component loadings.
Personally I would also suggest eliminating option #9 entirely, since Ive always felt it inappropriate to include the cubic terms when asymmetric load behaviour is already being modeled by other terms. I made this point in my contribution to the original document, (end of final paragraph of section 3.1.1):
In particular, when the linear and quadratic asymmetric load terms are included there is an argument to be made for elimination of the cubic terms, since their primary purpose is to provide better modeling of the balances behavior over the full range of loads, from negative to positive. Since the presence of asymmetric load terms effectively separates the positive load and negative load behavior, the cubic terms should perhaps not be included unless the bridge outputs exhibit definite evidence of cubic behavior for a particular direction of loading.
Unfortunately that caution seems to have been a victim of the stampede to embrace the 6x96 matrix as a panacea, but in fact it has since been pointed out* that there are sound reasons to suggest that certain terms in the 6x96 math model should not coexist. Several balance calibrations that I have been involved with have provided evidence that supports this observation. Given all of that, I wonder it its really appropriate to continue to recommend #9 , the 6x96 matrix, as a specific option in any revised version of the document?
* Calibration Designs for Non-Monolithic Wind Tunnel Force Balances. Paper by Johnson, Parker, Landman presented at the International Balance Symposium meeting at NASA Langley, 2010.
3. SUBJECT: Percent Contribution (fourth revision)
I'd be really interested to learn the origin of this proposal since it's exactly what I've been doing for ages! In 2002, at the time Stan was in San Diego for some of the calibrations in the AEDC balance manufacturer comparison project, I learned that he too used the same procedure. And I remember discussing the approach with Pat Schumacher at an IBTWG meeting in 2001, and then sending her a spreadsheet showing the matrix re-formatted that way, for one of the Ames balances she had had calibrated at FMS. So I can see that the idea might have originated with Stan, or perhaps as a result of my sample spreadsheet being distributed at Ames, but perhaps it originated in some completely different way. In any event, I'd be very interested to hear what it was that has prompted this particular input from Norbert at this time.
I'm completely supportive of including a description of this empirical, and somewhat subjective, assessment method in a revision document, but I question if it needs to be described in such elaborate detail - Norbert's style again. The essence of the process would be adequately described by the first sentence if the phrase in red italics were to be appended to make it read thus:
"The percent contribution is defined as the contribution of each term of the regression model of the gage outputs to the total predicted value, expressed as a percentage of the contribution of the principle diagonal term, where all contributions are evaluated at the rated capacity of each load or load product."
I think paragraphs 1, 2 and 4 have some content that's worth retaining, but in my opinion all of paragraph 3 is unnecessary padding that contributes little or nothing. In fact I think spelling out those 11 equations just serves to make the process look complicated when it's really trivially simple. What's difficult to understand about evaluating the term contributions and expressing their ratio as a percentage? Plus, when you come right down to it, why is "CAP", (which represents the component capacity - an absolute value), shown in the equations in both 'signed' and 'absolute value' form, both alone and in the 2-load products? If it's felt that there is a need to show equations, then really there are only 4 required - Eq 3.1.10 plus one for each of the THREE "groups" of terms in Eq 3.1.2, i.e. Fj, Fj.Fk and Fj3; any additional equations are superfluous.
Finally, in the last (4th) paragraph it's stated that the 'percent contribution' process is able "... to only identify a subset of the statistically insignificant terms ...", and there seems to be the inference that any model obtained in this way would be inferior to one evaluated with objective statistical methods. From practical experience I know that it's possible for math models determined empirically to equal or exceed the performance of BALCAL's 'statistically derived' model - in 3 out of 3 cases I have examined this was true! So I don't really think the empirical process is at any great disadvantage in normal experimental situations, and the advantage that it's easily understandable is a very real and significant one. Norbert quotes a significance cut-off value of 0.05%, but there is no reason why a lower value could not be used if felt appropriate. In that case it could well be that the terms identified as insignificant by statistical methods, might turn out to be a subset of those identified by the empirical process, rather than the other way around! I would have thought that Norbert might have become less dogmatic about setting empirical thresholds following the Canard balance debacle. To quite an extent that was the result of his insistence on setting too low a VIF threshold, and thus causing BALCAL to reject a perfectly valid math model! Indeed I see in the #1 proposal document that he is quoting a VIF threshold of 10; for some completely functional balance designs that I have worked with, that threshold would definitely result in some important terms being improperly excluded!
Thank you Dave for giving me the opportunity to review these proposals; though no longer really active in the balance community Im still interested in developments. Should you feel that anything Ive said here, or in the other attached documents, would be of interest to others at the upcoming IBTWG-2 meeting, please feel free to use any or all of it as you wish.
With very best wishes for Christmas and the New Year, Robin.
" $
[] !!o""%%''L(M($*Q,,,@-R-l.'/(/\/33444O5R7c7Q8V888889f9x>>@
6B*CJB*CJB*CJ>*CJ6CJ6B*
6B*CJ5CJCJH*CJG
9g N*n"o""&(#*S,l.'/(/\/24x
9g N*n"o""&(#*S,l.'/(/\/24R58??@@@#4R58??@@@x/ =!"#$%
[$@$NormalmH <A@<Default Paragraph Font4O4DefaultB*CJhmH nH <R!
o
.<6@%4@&(@'8?f;=pr}Z\
ho# , M!S!!!####w$|$$$%%**++,,.404l4n4{4}444y66<gj,Ekn#-!!**//0Q1]5f5m:n:X;Y;<<<RobGalZC:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\AutoRecovery save of Comments on Norberts proposals.asdRobGalZC:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\AutoRecovery save of Comments on Norberts proposals.asdRobGal,N:\IBTWG2\Comments on Norberts proposals.docRobGal,N:\IBTWG2\Comments on Norberts proposals.docRobGalZC:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\AutoRecovery save of Comments on Norberts proposals.asdRobGal,N:\IBTWG2\Comments on Norberts proposals.docRobGal,N:\IBTWG2\Comments on Norberts proposals.docRobGal,N:\IBTWG2\Comments on Norberts proposals.docRobGal,N:\IBTWG2\Comments on Norberts proposals.docRobGal,N:\IBTWG2\Comments on Norberts proposals.doc@;;S;;<P@GzTimes New Roman5Symbol3&zArial"qhMZm0h!>0&<
9/Dec/2011RobGalRobGalOh+'0p
,8
DPX`h9/Dec/2011/DeRobGal0obGobGNormal.dotRobGald20GMicrosoft Word 8.0@j2@V۶@0#0՜.+,D՜.+,8hp|
/h&<19/Dec/2011Title 6>
_PID_GUIDAN{C2ED11C7-5DB2-402C-9DD2-4D6D384BE251}
!"#$%&'()+,-./013456789;<=>?@ADRoot Entry FP<'#F1Table*WordDocumentRSummaryInformation(2DocumentSummaryInformation8:CompObjjObjectPoolP<'#P<'#
FMicrosoft Word Document
MSWordDocWord.Document.89q