Models are used to provide a representation of a system, usually as a learning tool. Architects build miniature models of buildings and commercial centers, so their clients can see what they are going to build. Inventors build models of their inventions as part of explaining what’s special about their new creation to the patent office. Models may be physical or mathematical, hardware or software. The goal of building and using the model is to understand the underlying structure of whatever is being modeled, and also to be able to understand how different forces will affect the structure’s behavior. In the case of model buildings, we may want to know how the building will be affected by strong winds. In the case of a mathematical model, we may be seeking to understand how a company will be affected by economic changes. The model is always created with one or more specific purposes in mind. What we want to know will strongly influence how we create the model.

The models we have chosen to discuss are:

  • The books of a company, as a model of the company’s performance over time.
  • A supermarket’s business model.
  • Models of stock & bond prices.

Using these models, we will discuss using the model to forecast the future, which requires a combination of history and extrapolation. Forecasting introduces assumptions, both stated and unstated about the system that’s being modeled.

Finally there a discussion of extending the model for econometric or future pricing of stocks & bonds. Together with some questions which should be answered by the model builder and the model builder’s management, as this author believes that the model builders on Wall Street are arrogant & reckless, and that their management cannot claim innocence, saying “We didn’t know the model wouldn’t work.”

This essay is written in an effort to protect the public, who seem to be forced to trust the people who create the economic models, and claim that the models always tell the truth and are accurate. We focus on the processes necessary to ensure a model is both precise and accurate. There’s a brief description of the math used at each stage of modeling, in terms that are easily understandable. We invite you to look at our words, in the hope that this paper will make it easy for you to see how models work, and why they are always limited in their ability to reflect the past or to predict the future.

Let’s start with the definition of a model:

A schematic description of a system, theory, or phenomenon that accounts for its known or inferred properties and may be used for further study of its characteristic.

A model is something created in order to be able to study something larger; the model may be physical or mathematical, and may be simple or complex. Models are built because they simplify studying something bigger. The modeler chooses the type of model he wants to build, depending on what he wants to learn from the model. His choices determine how the model works, how well it works, and its limitations. No model EVER contains the same amount of information as the building or plan which inspires the model. The goal of the model is to simplify studying certain aspects of the building or plan, so facets of the original which do not seem to be important to the study are left out of the model. Leaving these things out simplifies the model, but also defines the model’s limitations.

In the definition above, please note the words “theory,” “inferred,” and “further study” in the definition of these models. When building models, we need to continually check on the accuracy of the model; otherwise, the model will be useless, not fit for its intended purpose. The words “theory,” “inferred,” and “further study” in the definition above show that the modeling process should heavily reflect the scientific method, because we want the model to be as accurate & reliable for its purposes as possible. To learn about the reliability of models, we need to understand the scientific method. Here’s a short description of how the scientific method can be used to build a model, from Wikipedia:

  1. Use your experience: Consider the problem and try to make sense of it. Look for previous explanations. If this is a new problem to you, then move to step 2.
  2. Form a conjecture: When nothing else is yet known, try to state an explanation, to someone else, or to your notebook.
  3. Deduce a prediction from that explanation: If you assume 2 is true, what consequences follow?
  4. Test: Look for the opposite of each consequence in order to disprove 2. It is a logical error to seek 3 directly as proof of 2. This error is called affirming the consequent.

The scientific method can never absolutely verify. It can only falsify. One can never, ever, state a model gives CORRECT results.

Notice that the scientific method isn’t perfect; you can never prove that something is perfectly true; you can, however, itemize the cases where the conjecture (step 2) fails. A corollary of this is that even models built on the scientific method will never be perfectly able to model future events. They can’t even perfectly model past events. The following example is one with which we are familiar.

A Fairly Simple Model: Company’s set of books

Let’s discuss a model with which we have much experience, a set of books for a company. The chart of accounts is the structure of the model for the company, and the individual transactions are the details of the model. Accounts use two mathematical operators, addition and equality, and from the math perspective, a set of books uses very simple math.

Accountant and bookkeepers go to great lengths to eliminate errors in these books through the double entry, balancing and reconciliation processes. These processes are the application of the scientific method to the accounting model of a company; you build the model details (using addition), describe a conjecture (it balances using equality), and to test it (equality and reconciliation).

Any error, imbalance in the books, makes the results of the model invalid, and results in much work by the accounting staff to correct the model, and may result in making specific assumptions, (called adjustments,) to make the books balance.

The Set of Books for a company is a very limited model, but it is still complex to manage. Even though the model makes absolutely no predictions of the future, it still take a lot of work to verify and keep the model accurate. There are NO unknown variables! All the variables, the items in the chart of accounts, are known, and all the relationships among the variables — branch, business division, geographic locations, bills of materials, sales people, commission, etc. are fully known and defined.

Still we get errors and fraud in accounting! Even with a well-known model, based solely on recording the past! The model even has built-in protections: excellent procedures for entering correct (clean) data, error-checking (double entry bookkeeping, reconciliation to bank accounts, and auditors who have the responsibility of keeping the accounting honest, the model still suffers from errors and omissions. Even with a relatively simple financial model, which does not pretend to be able to forecast the future, as some models claim to do.

Models that Purport to Predict the Future (Forecasts)

The next step that businesses take with their Model of the Company is budgeting and forecasting typically for the next year.

Budgeting is the process of making assumptions within strict limits on the costs, for known activities for the coming year. This includes budgets for existing personnel and locations, and budgets for expansion, new locations, personnel, products and markets.

Forecasts are typically estimates of future revenues from existing personnel, locations and markets, with the addition of estimates, based on a good knowledge of what is feasible in the business for new personnel, locations, products and markets.

Businesses rarely use basic statistics (such as standard deviations or variances,) to express future revenue in a range, and the concept of a significance check to determine that the population of customers or employees is actually consistent with the statistical assumptions is almost completely unknown. The model is as simple as possible.

Forecasts use averages of sales per sq ft, average revenue per salesperson, estimated addressable market as assumptions and estimates of the future revenue. Such forecasts contain an assumption, that averages are a good measure of business performance. Unfortunately, forecasts based only on averages are always wrong, because averages are insufficient to measure the success of the business. A business never operates at an average level; it always operates within a RANGE; sometimes above average, sometimes below. When the business model relies on averages alone, it actively omits additional information which would allow it to represent the business more accurately. When a business model truncates data this way, it always fails to give optimum information to the business. The consequence can be as severe as business failure, or merely a growing reputation for poor customer service. Truncation of data in a model is often the result of assumptions built into the model. Sometimes the assumptions are clear and expressed; sometimes they are hidden, and sometimes even the modeler is unaware of the hidden assumptions.

Assumptions are somewhat tricky. IBM in its heyday used to state that IBM did not enter niche markets. That statement hides an assumption, that there are other than niche markets. An alternative assumption might be that all markets are niche markets; some niche markets are larger than others. In this assumption, there are only niche markets, the distinguishing feature being that some niches are larger than others, as IBM finally discovered.

Other items may affect the efficacy of a given model, and these items may not even be under the control of the business. These are external events and forces. One example of such an event might be the entry of a competitor into the market. Suddenly, the economic outlook for a business may change, as it did when many small town businesses found Wal-Mart setting up shop and taking their customers. This lesson teaches us that no model will be able to predict everything.

Even established businesses, with a strategy that works well, can fail when they enter a new market area. Often, in terms of models, the reason for the failure is that the model did not conform to the actual business situation in the new market area. There are a number of reasons why the model may fail; let’s consider an actual example, and analyze it for more information:

Ralph’s Supermarket

In my area, a Ralph’s supermarket, selling its standard range of products failed. In its place another supermarket is successful. The difference between the success of the new supermarket and the failure and closing of Ralph’s is easy to understand. The area’s demographics changed, from majority Caucasian to majority Asian.

Asians in this area do not eat Caucasian processed, canned or frozen foods. Their food is fresh, their vegetables different, and their fish is swimming in a tank when bought. Ralph’s produce was of little appeal to the Asians, and the store sale per square foot fell until Ralph’s closed the store.

The hidden assumption in Ralph’s business model was that the buying habits in the US are uniform, across the country. As a result of that hidden assumption, Ralph’s defined any significant negative deviation from their normal supermarket sales volume as a failure, indicating that the store should be closed. This is certainly one valid view of the business. Was there another explanation?

Another, better assumption is that the business cannot succeed if the product range is unappealing. This immediately leads to the question: “Who shops at supermarkets in this area?” Ralph’s assumption of a uniform marketplace was wrong. In this area, the majority of the population are of Asian heritage. The food they buy is not the same as the majority of Americans. Ralph’s hidden assumption in their business model caused them to ignore a huge local market, which might expand internationally to the places that these Asians came from. Effectively, a flaw in their model caused them to completely ignore a potentially huge business opportunity. Ralph’s supermarkets’ model FAILED, because the model had a hidden assumption built into it. Their model led them to ABANDON the market, instead of finding a way to expand their sales locally, and possibly internationally. There are about 2 billion Asians worldwide, and many local enclaves within the US.

Did Ralph’s know about its assumption? Given that they failed to capitalize on a new market opportunity, the answer seems clearly to be “No.” Ralph’s does not understand the assumption in its business model, and failed to expand into a market 10 times the size of its home market.

Hidden assumptions cause failures. This with a model that uses middle school math, addition, subtraction, equality, multiplication, percentages, division and maybe, summation of a series and square roots for basic statistics.

Complex Models of Stock or Bond Prices

We’ve already established that building, calibrating, testing, verifying, and using models is complex and difficult – even with simple models. We also established that both external events and hidden or ignored assumptions case invalid results.

Let’s take a team of very, very smart, well-educated mathematical geniuses and have them build a model to predict stock or bond prices in the future, using very advanced calculus, complex integration, second and third order differentiation, and some non-linear feedback. Then we’ll ask some basic questions:

  1. What assumptions do you know you’ve made?
  2. How do you verify the model’s results?
  3. What variables, factors, are included in the model?
  4. What are the relationships between the factors?
  5. Where did the definition of the relationships come from?
  6. Is this definition complete?
  7. How good are your data sources?
  8. What is the model’s sensitivity to changes in inputs?
  9. What is provably false about the model?
  10. How do you apply feedback?

These questions should suggest to you that any complex model must be thoroughly vetted before its results are believed. Moreover, any time its results seem out of harmony with common sense, common sense should prevail until the results are provably correct. It may not be possible to prove that the results are correct; in that case, use the model’s results as untrusted input in your decision-making; use it if the results support other data and if they agree with your previous experience. DO NOT use those results by themselves, to make any business-critical decisions. Remember, your model’s results may be precise, but they may not be accurate! If the smart people you hired to build the model cannot answer any of these questions, or do not give you convincing answers, do not trust your model to make decisions for you. Make your own decisions, remembering that the model has its limitations.

Wall Street’s model for credit default swaps and bundled mortgage securities was based on a complex model, and used advanced mathematics. Moreover, it had some built-in (but known) assumptions. One of these assumptions was that the math behind the model would not be used when unknown external events or forces could influence the model. These unknown forces could invalidate the model, without giving any warning. Unfortunately, the genius behind the model used the mathematical lemma without giving due regard to the warning about external events. The result? Their model failed, catastrophically, and took the world’s economy with it.

Whenever a model is used in the decision-making process, it is incumbent upon the decision-maker(s) to know the limits of the model, to be certain that the model can be used in the present circumstances of the business, and to know that the model is thoroughly tested on a regular basis. You need to know more than that the author has signed off on the model; anyone can build something that they cannot successfully debug. That doesn’t prove anything. You need to have confidence in your model, based on tested proof that it is working to the best of its ability. Anything less, and you may be negligently responsible if something goes wrong.

Summary and Conclusion

Difference between Precision and Accuracy, or Precisely Wrong
As we have established, even if all the variables in a predictive model are well-defined, all the relationships between the variables are known, all the assumptions are stated, and the data for the model is the best that skilled professionals and managers can provide, forecasts may be precise, but are very rarely accurate. Note the difference between precision and accuracy: the result may have a large number of decimal places, indicating a very precise result. This does NOT guarantee that the output data from the model are ACCURATE, however. The results may be precise, with lots of decimal places, but still be completely wrong! For this reason, models need to be thoroughly tested, for both precision and accuracy.

How can you decide whether a model is good enough?
We hope, by this point, that you have a more cautious view of models than when you started reading this paper. Models can work well, but they require careful design, and maintenance in order to be useful. Models which have hidden assumptions, or which do not have enough data, or in which the data is wrong will still give you answers – but trusting those answers will lead you astray, sooner or later, and can even cause your business to fail. The model should also be as simple as possible (but not simpler, as Einstein said). A simple, complete model can give good data for your decision-making process. Knowing that it is both simple and complete requires thorough testing and regular re-testing. Models tend to be dynamic, not static. When circumstances change, you need to update your model. And remember: Keep It Simple!

On the other hand, if your model is complex, using advanced mathematics, lots of variables and input conditions, and requires significant computing power to give an answer in a reasonable time, you need to pause before acting on its results. There are many places in a complex model where an error can creep in, and if the data or the model have errors, the output from the model will have errors, possibly magnified! Simple models are difficult to build and debug; complex models add another order of magnitude (or two) of difficulty, and their output should always be considered suspect. Do not bet your business or your personal fortune solely on the output of a model. Sooner or later, you’ll lose!

Would Wall Street bet your money on faulty models?

We’d assert, and here’s our $1,000,000 bet, Wall street’s complex econometric models do not have all possible variables defined, the relationships among the variables are not well known, and there major are unstated assumptions (one being that the economy is in a steady state).

We bet our $1,000,000 that in any institution using complex mathematical models that we could invalidate the model in less that one working day for one or more of the reasons above. Of course, if they wanted to take us up on this bet, they would have to be willing to discuss their models with us in detail. The first thing we would do is ask them the ten questions listed above. And we’d require proof of their assertions.

How is it then, that Wall Street and the financial industry believes it can “mark asset values to model” or use models to predict the economy, or use models to predict stock or bond prices?
These assertions smack of arrogance by the builders of the model, and gross negligence by their management, who promote the use of these models for managing money, without asking and disclosing the answers to the questions above. Especially when these models are non-transparent, use complex and advanced math, and are fraught with potential implementation errors, with no oversight by their business’s auditors, who probably wouldn’t understand the model even if it were explained to them. These models are VERY complex.

Accounting has the Financial Accounting Standards Board, or FASB, which defines process, and business accounting methods. Businesses have auditors to verify that these standards are correctly applied by the business. How is it that the Money Managers do not have to disclose disclose how they verify their processes, and there is no oversight by auditors? Lack of transparency and independent oversight is a recipe for misrepresentation, negligence and fraud. There is not nearly enough regulation of Wall Street and the banks; that’s why they bet with your money, and if their bet fails to win, YOU LOSE!

Loose Morals

And the final moral kicker for the Wall Street types:

Will you bet every penny you possess on this business model? If not why not?

If you won’t bet your money, what right do you have to bet with other people’s money?