K7—, thank you for contributing your excellent, and insightful, analysis and recommendations for improving! -JJS
TEOTWAWKI Readiness Exercise – 2014
After Action Report
The TEOTWAWKI Readiness Exercise – 2014 (TREx-2014) was a national-level exercise that included as many as 1,200 active and passive members in AmRRON together with an unknown number of non-member passive radio listeners. Responsibility for responding to the catastrophe during the conduct of the scenario rested with each individual family or neighborhood. There was no organized authority conducting the survivalist response to the simulated crisis.
The primary purpose of the exercise was not to wring out problems in household disaster preparations. Rather, the purpose of the exercise was test the ability of the participants to make survival decisions while under duress and while attempting to honor pledges (to relay news messages) to people outside their kith and ken.
Duress was imposed by having the participants read a series of 80 threatening news stories over a full 10 days, at all times of the day and night, leading up to a point in time at which the power and communication grids failed. Once the grid went out, the participants were then forced to scrounge for post-outage news from shortwave radio. Those who could receive radio messages during the grid outage were fed an additional (post-outage) 32 news reports that further revealed the nature of the crisis.
Participants were required to reach their own conclusions, from the news report messages alone, as to the nature of the threat, its likely duration, the urgency of responding to it, how to respond, and how to save neighbors and family.
A secondary objective was to establish and operate a nationwide emergency radio network for the sole purpose of exchanging news reports, independently of any commercial or government agency, operational structure, pre-agreed protocols, or signaling channels.
Event Description and Timeline
The grid did not fail all at once. First, the power failed for several hours, was then restored, and finally failed permanently four hours after it was restored.
On the Internet, social media sites were taken down by a cyber attack. Reverse 911, cell phone and landline telephone systems slowly fell apart before they stopped working completely. Satellite Internet service returned for a few hours in the middle of the scenario, then even that failed permanently.
Broadcast TV and radio likewise degraded and then failed when backup generators ran out of fuel or solar arrays were looted and stolen. Simulated newscasts were podcast from www.amrron.com until all broadcast services shut down. The news stories up through Thursday morning were posted in text form on the AmRRON website.
The following point cannot be emphasized enough: There was no explanation – no interpretation – of the 112 news reports issued over the course of the exercise. Each individual had to separate what was important from what was “noise” all on his or her own.
For example, at no point in the exercise was martial law explicitly declared, at no point in the scenario was there an announcement by the White House that ten American cities were to be quarantined. But all of the news stories pointed towards both. It was left up to the individual to receive the news stories, to understand them, to try to form a picture of the world situation from them – without any central authority – with no authority available to confirm or refute all those individual assumptions.
The timeline is in two pieces: The period before the near-complete loss of communications (Friday afternoon) and the following period to the end of the exercise at Sunday noon, August 24, 2014.
The exercise started at 2330Z (16:30 hrs. Pacific Time) on Wednesday, August 19th, 2014.
Exercise Period #1: Multiple Crises Building Worldwide and at Home (80 news reports)
The following news themes were played out over the period August 12th through 1900Z on August 22nd:
- Central Banking Crisis leading to a run on the banks, including Bank of America, and a shutdown of banks in the US, with no ability of customers to withdraw money for daily essential purposes or otherwise. Most of this was actual news taken from financial news headlines. Banks collapse in Greece, Portugal and other countries.
- Government Shutdown of Means of Dissent by the government taking control of people’s cell phones and shutting them down in order to prevent people from sharing news. Again, these were mostly current news stories, since the California state legislature passed a law to permit just such shutdowns while the exercise was underway.
- ISIS Gains More Territory and Threatens the US. Again, these were almost all real news stories from current press reports.
- Riots Spread from Ferguson, Philadelphia and Los Angeles to New York and Other Cities. About half of these news stories were real and the remainder were fabricated.
- Power System Explosions in Indianapolis. A real story (not fabricated).
- Cyber Attacks on UPS and Union Pacific Railroad. The cyber attack on UPS was real. The UP Railroad cyber attack was fabricated. It led to a shutdown of half the nation’s rail traffic. This, in turn, caused coal-fired power plant shutdowns, gas-fired power plant shutdowns due to fallout from the shutdown of the coal-fired plants, and massive food shortages.
- Russia Invades the Ukraine. US sends 600 troops to bordering countries and even more troops arrive in Bulgaria (real news).
- Ebola begins to spread to Europe.
- The Banking Crisis Causes a Halt in International Trade. Ships cannot be unloaded in the harbors, trains and trucks don’t move, people are starving and fuel is gone.
- Cyber Attacks Increase. Unknown forces take down Facebook, Twitter, Community Health Services, Inc. and many other sites. Amazon cloud services are shut down by hackers. Some of this was actual news.
- FEMA Announces Food and Fuel Confiscation and Redistribution to Those Who Submit to FEMA Concentration Centers.
Things accelerate in the final 36 hours prior to the grid outage:
- The Secretary of State and Director of DHS Resign.
- Delta Flight 1082 Struck by ISIS Missile in California and Crash Lands at Moffett Field, California. US National airspace is closed. Stock market crashes and is closed. All commerce halts. Shelves in grocery stores are empty.
- The first US Ebola Case Mentioned in the Press.
- Government Imposes a News Blackout of Many Stories, including the crash of flight 1082, spreading rumors of Ebola cases in the US, and terrorist actions on US soil.
- Power Outages Spread Across the US.
- DHS Given Total Authority Over All Other Domestic Agencies, Run by White House Political Advisor Valerie Jarrett.
Exercise Period #2: TEOTWAWKI
Period 2 begins with the loss of electrical power nationwide at noon Pacific Time on Friday, 8/22/2014. At first, 911, reverse 911, emergency responder radios, local TV and broadcast radio services still operate. Cell phones, cable, Internet and landline phones are off the air. Several hours later those, too, failed.
From this point forward, the news stories available to the participants differ from one person to another. No single person received all news stories. Those who were dedicated amateur radio operators were able to collect around 25 to 28 of the 32 news stories. We don’t believe that any person collected all 32 of them.
The news stories originated not from one central source or news agency, but rather from the individual network members themselves. The stories were either (fabricated) first-hand accounts or stories passed along from a simulated intercept of emergency service radio networks.
There is, therefore, no universal synopsis of events in Period 2. Each person must conclude for themselves what the news meant and implied to them.
Appendix A is a list of the news stories that were captured by K7- – – in Northwestern Nevada. The reader of this report is encouraged to read them one-by-one, to see if a coherent picture of the domestic situation can be pieced together. In fact, that cannot be done with any great certainty.
Please note that, in Appendix A, the stories are printed in the time order that they were received, not at the time the story was first created or first transmitted. The underlying events should not be assumed to correspond to the time order of receipt of the messages. In fact, that is contraindicated.
Observations – Psychological
There were an uncounted number of people who could not grasp the essence of the exercise – making life-or-death decisions in the presence of little and confusing information – and who kept reverting to a training exercise model for TREx-2014. In a training exercise, people are instructed on what to do, then repeatedly practice executing those instructions until they can operate autonomously when presented with the same or a similar situation. TREx-2014 was decidedly not a training exercise.
Those who were stuck on the training exercise mental model would “break out of character” (violate the rules of the scenario) to send emails during periods of time when there was supposedly no Internet service, expressing frustration that they were not given a description of the scenario itself, in a story-book fashion, with a beginning, a middle and an end, in advance. They didn’t understand the concept of deducing the story from little news items that they catch along the way.
The inability to function in an environment of real-world chaos could certainly be fatal to them someday. Helping those people to improve their deductive and decision-making skills is critical to survival, and it was concluded that more disaster exercises could be justified on that basis, alone.
The natural human instinct to take care of other members of the tribe became dominant among a large number of participants much earlier than expected. TREx-2014 had pre-published a timeline that specified the news-sharing (radio) network would start operation at noon on Friday, August, 22nd. This was to have been preceded by ham operator practice radio networks on Wednesday and Thursday. In actuality, many if not most of the practice session participants operated as if TREx had already started by Wednesday night, when a few started passing news messages to one another.
This instinct to join a self-protective group and to answer a call of duty to the other members of the group must have had strong psychological underpinnings. There was a strong need to not be alone during a time of crisis.
The compunction towards group identity in a real crisis might very well indicate that the transmission of news to members of a population under stress is of such high importance that it could perhaps eclipse, at least for a short time, the search for food, water and shelter.
From this, it is a natural conclusion that additional events like TREx should be conducted to give an ever-widening circle of people the opportunity to participate. A second conclusion is that, in every family group that has an active radio operator working to receive and retransmit news, the radio person needs to be protected from many other emergency preparedness chores. If those other chores require physical strength repeatedly or over an extended period of time, it might be better if the radio operators were the physically-weaker of the clan.
Most operators were self-reliant, collaborative, and cooperative in their approach to the mission. There were a few, however, who, out of nervousness or for some other reason, became glib, or angry, or felt slighted in some way and began to act in a juvenile, narcissistic way with other operators. This will always occur in a group situation and it placed extra stress on the operator network, as well as disrupting the message flow to the extent that some messages were lost.
Another natural conclusion from this exercise is that future exercises should attempt to further increase the operator stress levels, so that they can find solutions to high-stress conditions under simulated conditions, without actual risk of physical harm to themselves or others.
As with any situation that includes a call-to-action, whether it be a vacuum cleaner sales pitch at a home and garden show, or a call to arms in a civil war, the 3% rule seems roughly accurate: Only 3% will answer the call-to-action. That was most-evident in TREx-2014. Many individuals and groups said they wanted to participate and committed to participate, but never showed up for the networks, even though the real, actual news during TREx-2014 was describing real crises almost as threatening as those used in the simulation.
Low participation has historically been proven the case for any volunteer event. What makes TREx different is that many people now know about the AmRRON/ TAPRN and Channel 3 Project networks and know radio operator/ members in their neighborhoods. The no-shows for TREx-2014 will probably come forward to want to become actual, volunteer operators in a real crisis, and therefore the experienced operators need to reach out to them to help them prepare, to encourage better participation in future exercises, and to start up and operate local networks persistently, until the membership builds.
A more-recent psychological phenomenon is the belief that writing something on the Internet about a problem is the same as solving the problem. Fig. 1 expresses the problem better than can I.
Fig. 1. People Confuse Broadcasting a Problem to Others with Solving the Problem
TREx was really not designed to test for this phenomenon in a way that could be made visible to anyone outside the immediate home of the exercise participant. The question to be put to the pool of participants would be something similar to this:
“Did there ever come a time during TREx-2014 where, based on your interpretation of the news as you were receiving it, that you formulated a plan of action in your mind, but instead of taking that planned action, you posted something about the problem on the Internet (e.g., Facebook) instead?”
This issue can really be addressed only by self-examination after the exercise is complete, and to that end, it would benefit the participants to have an after-action self-assessment template prepared prior to the next exercise.
Observations – Network Self-organization and Regulation
Those not familiar with shared-medium networks, such as a party-line telephone, the speaker’s podium at a city council meeting, or a radio network that shares a single frequency, may be unaware that certain control protocols have evolved to reduce traffic “collisions” (two or more people trying to talk at the same time). The most-common of these is perhaps Robert’s Rules of Order.
All of these network control mechanisms require a single, authoritative network control person or automaton. The controller can operate in one of two regimes: In an open network, or in a network closed to anyone not previously admitted to the membership roster.
Either way, the protocols all take similar form: First, the central control person, called the Network Control Station (NCS) for amateur radio, asks people to “check in to the net.” Check-in transmissions are very brief. Because of this brevity, collisions on the airwaves occur much less frequently and with less impact during the check-in period than they would with longer transmissions. Having compiled a list of all those who want to speak, the NCS then goes through the roll, usually in order of recognition, and calls each checked-in person to speak. After an individual has finished speaking, either the NCS or the next person in order is permitted to speak. This control method does not permit back-and-forth conversations to occur, since the NCS must recognize each speaker, one at a time, and no person is allowed to speak until called upon to do so.
For amateur radio, each community has organized formal networks of this kind that go by the name of ARES, RACES, SATERN, the North Valleys Horse Breeder’s Net, and so forth. They are all networks of pre-admitted members who have pre-arranged times and frequencies of meeting in the event of an emergency. In the Washington State forest fires of August, 2014, the ARES nets completely failed. Government functionaries commandeered the radio repeaters (without any legal authority to do so) and kept amateur radio users, whether ARES members or not, from using those frequencies. But, the government never used the frequencies either, leaving them idle the entire time.
This affirms the need for an extra-governmental network to serve at the local and regional levels in additional to the national level. The Channel 3 Project and the AmRRON / TAPRN VHF networks can fill this role.
At any level, and in particular at the multi-state or national level, there is no purely ad hoc radio network aside from the AmRRON/ TAPRN network.
By ad hoc, we mean a network that can operate without Network Control Stations (NCS) and without pre-arranged membership rolls. In a real emergency, no hierarchical structure can be assumed to exist, not even if backups to primary NCS stations have been assigned. But, in addition, at the multi-state and national level, the physics of radio propagation take over and it becomes impossible to predict or even to contemporaneously assign an NCS. You never can predict who you will be able to contact, due to propagation effects, and if you have contacted them, how long the conditions will last that will allow you to maintain a connection. This clearly makes a top-down net control scheme impossible.
So, the AmRRON / TAPRN shortwave radio network is designed as an ad hoc, mesh network without control channels. In practice, this means that individuals who need to communicate or propagate news stories must find one another, agree amongst themselves if more than two parties are involved as to who transmits their messages in which order, and others not needing to transmit must remain silent and listen.
They must also be able to gather around a common radio frequency at the same time, which will, from time to time, require the participants search other than pre-arranged radio frequencies if non-participating stations are occupying them. This actually happened in TREx-2014 at least twice. If nothing was heard on the pre-arranged frequency at the pre-arranged time, then people would “look around in the neighborhood” to see if they could find another AmRRON operator who was active. This actually worked fairly well.
In a real crisis, stations are operating on emergency power. They simply cannot have radios running around the clock, even if just listening. A document called the Signals Operating Instructions (SOI) contains a schedules of times-of-day and frequencies at which radio operators should meet.
There is one meeting opportunity each six hours on each frequency and transmission mode, voice or digital. This is an excessively-dense rota in a real emergency. Consolidation of the frequency plan is advisable from a workload perspective, but likely impossible for technical reasons having to do with the way radio waves propagate as a function of frequency, distance, ionospheric conditions, and thunderstorm activity.
As a result, there will be many instances where individuals will tune up at the right time and frequency, but find nobody else there. This caused significant frustration on the part of many people during TREx-2014, to the degree that they simply gave up early in the exercise and walked away from it.
The expectations of those individuals clearly derive from their experience with modern electronic communications, which have a near-perfect probability of making and sustaining a connection.
At the same time, AmRRON / TAPRN run twice-monthly practice networks. These are presently structured nets, with an NCS station. Some consideration might be given to simulating an ad hoc message-forwarding network such as that used for TREx-2014 instead of using a structured control network for those bimonthly practice sessions.
An ad hoc, mesh network over noisy physical carriers cannot be used for reliable broadcast of messages to all participants. Any attempt to do so will cause network congestion to spiral out of control and crash the net. This is a mathematical certainty.
The problem is solved by reducing the reliability and / or geographic reach of transmission of each message, and by conserving bandwidth to the maximum possible degree.
It should also be kept in mind that the vast majority of network participants are receive-only shortwave listening stations. Many times, they can only hear one side of a radio conversation. In that circumstance, if the intended receiver of a message in a conversation with the message sender does not repeat back the received message, the shortwave listeners will have no means to recover the message.
Repeating back all messages provide a good means of error detection and correction and serves the shortwave audience, but doubles the bandwidth consumption of each message sent. TREx-2014 clearly demonstrated that reading back each message was impossible with the traffic load offered, and within the transmission windows available.
This problem bears significant further study.
Aside from the readback problem, all other bandwidth conservation measures elicited through TREx-2014 are straightforward:
- Eliminate extraneous characters, such as header and footer markers
- Use easily-recognized abbreviations instead of spelling out common words
- Set an expiration time on each message, after which the message should not be relayed further unless the channel is otherwise vacant or idle. Since the messages convey news stories, most messages will lose any tactical meaning within tens of hours.
- Identify which stations are capable of reaching nearest-neighbors through Near Vertical Incidence Signal (NVIS) propagation vs. those which have antennas that emit signals near the horizon and are therefore suited only to very long-distance propagation. By sending a message from an NVIS and a long-haul (“DX”) station, the back-haul retransmission problem is avoided.
- For certain messages, such as situation reports, that can have a fixed format, invoke a fixed-format standard so that field numbers or field identifiers can be substituted for descriptive words
- Create message headers that include a code for where the news in described in the body of the message originated. A flood in Atlanta, for example, if of no relevance to a radio operator in Oregon who is fighting a range fire.
Observations – Radio Network Technical
This section is for knowledgeable radio operators.
- Network operated far better than engineering would suggest. In engineering terms, this is an “Ad Hoc Mesh Network,” which in theory should be completely unworkable for message broadcast dissemination. Real humans took all sorts of practical measures to create an effective network for broadcast message dissemination in spite of this fact.
- 600% more transmitter energy was required than planned. For K7—, his solar+battery radio power supply system was sized for an average of 2 minutes transmit per hour, whereas the actual usage was closer to 12 minutes per hour – a 600% difference. The primary cause was the need to relay message traffic. Additionally, most digital nets over-ran their time allocations by 100%, requiring power for a laptop computer continuously during that time, even when the transceiver was in the receive mode.
- Omnidirectional antennas are required. Use of beam antennas creates severe operational problems in an ad hoc radio network environment. It would be better to use very narrowband digital signals rather than beam antennas as a means of increasing link gain.
- Digital nets took 40 minutes to an hour vs. 20 minutes forecast. The voice nets, which were partially abandoned after the first day, cannot be removed from the transmission schedule because there are probably many shortwave radio listeners without digital demodulation and decoding capability.
- Every possible effort should be made to conserve network bandwidth. Messages that are likely to have information value for only a short period of time should have an expiration date-time-group applied by the originator, after which time people should stop attempting to relay them.
- Additional bandwidth could be conserved by using fixed-field messages for SITREPs and persons with emergency service, pilot, or military training could use widely-recognized abbreviations; for example, “MSG” vs. “MESSAGE.”
- Message forwarding protocol could be improved and bears a detailed examination. People with no traffic to send and who need no known relay of traffic to them nonetheless expend a lot of time and bandwidth “checking in” to the net and providing QSLs, and even trying to start QSOs on random and unrelated topics. Also, an areal effectiveness could be assigned to the underlying news items, or the message headers could contain a code indicating the origin of the news story. These changes would permit geospatial filtering, either by relay operators or by those requesting message relays, or both.
- Operator workload was on the edge of unworkable. General and Extra class operators are the bridge between HF, 2m and the Channel 3 Project operators, and consequently have a workload that is so high, that a real emergency situation will demand that they must have immediate (close) family support. If they are alone or must also take charge of household emergency duties, they won’t be able to perform well as an AmRRON operator. Remember: “Aviate, navigate, communicate,” in that order. Communications come last. Survival and planning for survival take priority.
- Many Relay Opportunities Were Probably Lost Due to a Lack of NVIS Station Participation. There is no way to quantify these losses in network coverage. Those who lived under a 20m skip zone were dependent on “back-haul” relay from DX stations.
 “The End of the World as We Know It,” meaning a catastrophic collapse of civil infrastructure and authority, coupled with interruptions of food supplies and communications.
 American Redoubt Radio Operators’ Network, see www.amrron.com
 In a real crisis that took down the entire national grid, the satellite Internet service would fail as soon as the terrestrial fiber optic network failed. This was a small defect in the scenario.
 Written by K7- – – in Northwestern Xxxxxxxx.
 Action Item #1: Create many more TREx events to increase the opportunities to participate.
 Action Item #2: Seek ways to increase operator stress level at the next exercise.
 Action Item 4: Create local training opportunities, set up meetings, create regularly-scheduled local radio networks, and persistently hold those activities, even if, at first, no one shows up.
 Action Item Number 6: Create a self-assessment questionnaire.
 Action Item #7: Create non-exercise ad hoc network practice sessions.
 Action Item #8: Define message forwarding and repetition rules based on geographic coverage constraints