Simulace.info - User contributions [en]

Agent-based computational economics

2012-06-20T21:59:09Z

Xsamm12: /* Possible learning types */

'''Agent-based computational economics''' or shortly '''ACE''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices. It works in following steps:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types other of learning algorithms suitable for use in ACE <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics]. For connectionist learning various types of [http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] configurations can be used.

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==References==
<references/>

Agent-based computational economics

2012-06-20T21:56:01Z

Xsamm12: /* References */

'''Agent-based computational economics''' or shortly '''ACE''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices. It works in following steps:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types other of learning algorithms suitable for use in ACE <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==References==
<references/>

Agent-based computational economics

2012-06-20T21:54:35Z

Xsamm12: /* Other computing methods */

'''Agent-based computational economics''' or shortly '''ACE''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices. It works in following steps:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types other of learning algorithms suitable for use in ACE <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:54:16Z

Xsamm12:

'''Agent-based computational economics''' or shortly '''ACE''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices. It works in following steps:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types other of learning algorithms suitable for use in ACE <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:53:54Z

Xsamm12: /* Possible learning types */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices. It works in following steps:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types other of learning algorithms suitable for use in ACE <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:53:22Z

Xsamm12: /* Learning */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices. It works in following steps:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:52:17Z

Xsamm12: /* Agent types and characteristics */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. In some cases even without learning capability, the agents can optimize or are able to generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:50:02Z

Xsamm12: /* Software and programming */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:49:06Z

Xsamm12: /* Possible learning types */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.
[http://en.wikipedia.org/wiki/Cobweb_model Cobweb model] is example of [http://en.wikipedia.org/wiki/Genetic_algorithms_in_economics genetic algorithm for use in economics].

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:46:21Z

Xsamm12: /* Equilibriums and attractors */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. These can than be pinpointed through generated reports, plots or through other available ex-post analytical tools. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:42:55Z

Xsamm12: /* Computational world models */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

First method acquires (<code>getWorldEventSchedule</code> current time from the world itself. Through <code>getMarketProtocols</code> agent can acquire valid protocol used for different kinds of interaction and negotiations between agents. Method <code>getWorldProtocols</code> can serve for other out of market interactions.

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:38:07Z

Xsamm12: /* Computational world models */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational worlds can composed of various agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:37:22Z

Xsamm12: /* Fields of application */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. From this application it is not far to the economic field. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments on European scale.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:36:02Z

Xsamm12: /* Resarch */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools<ref name=Tesfatsion /><ref name=Tesfatsion2007 />. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:35:26Z

Xsamm12: /* Qualitative insight and theory generation */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization<ref name=Tesfatsion /><ref name=Tesfatsion2007 />.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:34:51Z

Xsamm12: /* Empirical */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy<ref name=Tesfatsion />.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:34:30Z

Xsamm12: /* Resarch */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref name=Tesfatsion2007>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:33:21Z

Xsamm12: /* Qualitative insight and theory generation */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood. It can explain why there are some regularities persistent over time and why they remain while others disappear. Evolving agent world can be used to observe needed degree of coordination to establish institutions and attain self organization.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:25:31Z

Xsamm12: /* Methodological advancement */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:25:05Z

Xsamm12: /* Normative */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of that organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:24:28Z

Xsamm12: /* Normative */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Subsequent insights in social norms and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:22:22Z

Xsamm12:

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses [http://en.wikipedia.org/wiki/Agent-based_model agent-based models] or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:19:49Z

Xsamm12:

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework, are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:19:19Z

Xsamm12:

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:12:46Z

Xsamm12: /* Examples of real applications */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims reduced workload at one participating company by 3 people. Total time time need for indentification of a client and claim was reduced from 6 months to less than 2 minutes <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf.</ref>.

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:06:14Z

Xsamm12: /* Examples of real applications */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf </ref>

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well<ref name=agentlink />.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing for 25,000 people by the Chicago Housing Authority<ref name=agentlink />.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution<ref name=agentlink />.

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T21:05:16Z

Xsamm12: /* Examples of real applications */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

*An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf </ref>

*Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well.

*Agent technology developed by Agentis Software was used to manage the complex processes and changing business requirements involved in the challenging task of relocating residents during project to refurbish or rebuild housing
for 25,000 people by the
Chicago Housing Authority.

*Agent technology by Agentis Software was used in project to rebuild and renovate housing for 25,000 people by the Chicago Housing Authority. Complex processes and changing requirements which were part of difficult task of relocating the occupants were managed thanks to this solution.

'''upravit, přepsat!'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T20:51:43Z

Xsamm12: /* Examples of real applications */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref><ref name=agentlink>AgentLink, 50 facts about agent-based computing,http://www.econ.iastate.edu/tesfatsi/AgentLink.50CommercialApplic.MLuck.pdf </ref>

Agent-based application from Whitestein Technologies (Switzerland) is used for optimisation of large-scale transport. Vehicles are represented as agents in the system. These agents negotiate through auction-like protocol. Vehicle capable of cheapest delivery wins the auction. This way overall cost of cargo delivery and often combined distance travelled by all vehicles as well.

'''upravit, přepsat!'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T15:57:19Z

Xsamm12: /* Examples of real applications */

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

An agent-based system developedby Acklin (Netherlands)for international vehicle insurance claims <ref>http://www.agentlink.org/resources/webCS/AL3_CS_004_Acklin.pdf</ref>

Whitestein Technologies (Switzerland) has developed an agent-based application designed to provide automatic optimisation for large-scale transport companies, taking into account the many constraints on their vehicle fleet, cargo, and drivers. Although the agent solution accounts for only 20% of the entire system, agent technology plays a central role in the optimisation. Negotiations are performed automatically by agents, with each agent representing one vehicle, using an auction-like protocol, so that the vehicle
that can provide the cheapest delivery wins the auction, reducing the overall cost of cargo delivery and, in most cases, the combined distance travelled for all vehicles.

'''upravit, přepsat!'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T15:44:39Z

Xsamm12:

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T15:44:18Z

Xsamm12:

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"<ref name=Tesfatsion2006>Leigh Tesfatsion, Agent-Based Computational Economics: A Constructive Approach to Economic Theory [(pdf,253KB) http://www.econ.iastate.edu/tesfatsi/hbintlt.pdf], in Leigh Tesfatsion and Kenneth L. Judd (eds.), Handbook of Computational Economics, Volume 2: Agent-Based Computational Economics, Handbooks in Economics Series, Elsevier/North-Holland, the Netherlands, 2006.</ref>
''
ACE is therefore computational study of economic processes modeled as
dynamic systems of interacting agents

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T15:42:33Z

Xsamm12:

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models, often created in specialized software or framework are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents. ACE is therefore '' "a computational study of economic processes modeled as dynamic systems of interacting agents"
''
ACE is therefore computational study of economic processes modeled as
dynamic systems of interacting agents

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T15:32:38Z

Xsamm12:

'''Agent-based computational economics''' is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents.

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T15:32:21Z

Xsamm12:

Agent-based computational economics is branch of [http://en.wikipedia.org/wiki/Computational_economics computational economics]. It uses multi-agent models or simulations to model real world market or economic interactions between agents. Agents can represent institutions, firms, individuals or environment. Models are dynamic and allow introduction of heterogenous or random (if needed) behavior of agents.

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T15:10:53Z

Xsamm12:

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Examples of real applications==

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:31:03Z

Xsamm12: /* Computational world models */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents artificial institutions e.g. market or between agents and the world itself. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:28:23Z

Xsamm12: /* Computational world models */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf, Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003,
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:27:29Z

Xsamm12: /* Agent types and characteristics */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf , Journal of Simulation, Vol. 4, 2010, 151–162</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:26:40Z

Xsamm12: /* Agent types and characteristics */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous <ref name=North>Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation", Journal of Simulation, Vol. 4, 2010, 151–162, http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf</ref>. Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

File:Derivate-follower-basin-of-attraction.png

2012-06-20T14:16:34Z

Xsamm12:

Movement of a agent using Deterministic Reactive RL. By using derivative-Follower adaptation the agent stops prematurelly before attaining the highest profit/value.

'''Image taken from:''' Leigh Tesfatsion, Learning Algorithms: Illustrative Examples http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf

File:Derivate-follower-basin-of-attraction.png

2012-06-20T14:15:39Z

Xsamm12:

Movement of a agent using Deterministic Reactive RL. By using derivative-Follower adaptation the agent stops prematurelly before attaining the highest profit/value.

Image taken from: Leigh Tesfatsion, Learning Algorithms: Illustrative Examples

Agent-based computational economics

2012-06-20T14:13:49Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image on the right shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Because of using simple Derivative-follower adaptation<ref name=TesfatsionLearning /> agent stops when profit level start's to fall, which is in this case too soon. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:10:20Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Agent can for be for instance attracted by different basins of attraction while using different learning algorithms. Image below shows how agent scale the profit curve using deterministic reactive reinforcement [[#Learning| learning]]. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:03:10Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:02:47Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

[[File:derivate-follower-basin-of-attraction.png|thumb|right|Agent stops too early in a basin of attraction missing the highest attainable profit]]
[[File:Příklad.jpg]]

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:02:26Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

[[File:derivate-follower-basin-of-attraction.png|thumb|left|Agent stops too early in a basin of attraction missing the highest attainable profit]]
[[File:Příklad.jpg]]

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:01:22Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

[[File:derivate-follower-basin-of-attraction.png|thumb|middle|Popisek]]
[[File:Příklad.jpg]]

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

Agent-based computational economics

2012-06-20T14:00:40Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

[[File:derivate-follower-basin-of-attraction.png]]
[[File:Příklad.jpg]]

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm

File:Derivate-follower-basin-of-attraction.png

2012-06-20T13:59:49Z

Xsamm12: Movement of a agent using Deterministic Reactive RL. By using derivative-Follower adaptation the agent stops prematurelly before attaining the highest profit/value.

Movement of a agent using Deterministic Reactive RL. By using derivative-Follower adaptation the agent stops prematurelly before attaining the highest profit/value.

Agent-based computational economics

2012-06-20T13:56:34Z

Xsamm12: /* Equilibriums and attractors */

==Resarch==
Main pillars of ACE resarch according to [http://en.wikipedia.org/wiki/Leigh_Tesfatsion Leigh Tesfatsion].<ref>Leigh Tesfatsion (2007) Agent-based computational economics. Scholarpedia, http://www.scholarpedia.org/article/Agent-based_computational_economics</ref> <ref name=Tesfatsion></ref>

* Empirical
* Normative
* Qualitativ insight and theory generation
* Methodological advancement

===Empirical===
This area stands for explaining possible reasons for observed regularities. This is achieved through replication of such regularities using multi-agent models. This approach allows to seek causal explanations thanks to bottom-up modelling of simulated market or economy.

===Normative===
ACE can help to increase normative understanding, ACE models can serve as virtual test field for different policies, regulations and can simulate many different economic scenarios. Insights in social nomrs and institutions can help to explain why there are some persisting regularities in markets. Another aspepct is relationship between environmental properties, organization structure and performance of this organization.
<ref>Tesfatsion, Leigh. “Agent-based computational economics: modeling economies as complex adaptive systems.” Ed. Leigh Tesfatsion & Kenneth L Judd. Information Sciences 149.4 (2003) : 262-268. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.4883&rep=rep1&type=pdf</ref>

===Qualitative insight and theory generation===
Through ACE approach, self-organizing capabilities of decentralized market systems could be understood.

===Methodological advancement===
representation, visualization, validation,
ACE seeks the best instruments and methods to study economic studies using computational experiment. Important aspect is whether data produced by such experiments are in accordance with real-world data. In order to achieve this methodological principles need to be developed as well as Programming, visualization and validation tools. For more information see [[#Software and programming |Software and programming ]]

==Fields of application==

One of the first major applications of multi-agent models in social sciences was famous [http://en.wikipedia.org/wiki/Sugarscape Sugarscape] model by Epstein and Axell. ACE can approach can be applied to rather simple [http://en.wikipedia.org/wiki/Double_auction double-auction] market models or two-sector trading worlds. ACE is also used in various complex market simulations like tourism, digital news or investments. ACE can also help to analyze the impacts of various policies and regulations for example effect of deregulation on an electric power market
<ref>
Cirillo R. et al (2006). Evaluating the potential impact of transmission
constraints on the operation of a competitive electricity market in
illinois. Argonne National Laboratory, Argonne, IL, ANL-06/
16 (report prepared for the Illinois Commerce Commission),
April. http://www.dis.anl.gov/pubs/61116.pdf
</ref>
<ref>
Charles M. Macal and Michael J. North, "Tutorial on Agent-Based Modelling and Simulation" [http://www.econ.iastate.edu/tesfatsi/ABMTutorial.MacalNorth.JOS2010.pdf PDF,359KB], Journal of Simulation, Vol. 4, 2010, 151–162
</ref>.
More complex models are capable of simulating whole economies with all necessary aspects as financial, household or job markets while maintaining homogenity of agents. Example of this is the [http://www.eurace.org/index.php?TopMenuId=2 EURACE] project. Models like this enable what-if analysis and policy experiments.
<ref name=Tesfatsion>
TESFATSION, Leigh. Agent-Based Computational Economics: Modeling Economies as Complex Adaptive Systems. 2010-03-24, [cit. 2012-06-18]. http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
</ref>. There are also applications to model economic behaviour of vanished civilizations
<ref>
Kohler TA, Gumerman GJ and Reynolds RG (2005). Simulating
ancient societies. Scient Amer 293(1): 77–84. http://libarts.wsu.edu/anthro/pdf/Kohler%20et%20al.%20SciAm.pdf
</ref>

==Computational world models==
[[File:AMES network.png|thumb|right|Agent hierarchy used in AMES framework]]
Computational world is composed of many agents, some of them can act on their own, have learning capability and memory. Others represent rather reactive elements of the world such as technology or nature. Some agents can be passive like house or patch of land. Composition of agents is also possible, music band agent can be for instance a composition of agents playing musical instruments. Agents are therefore ordered in hierachy as shown on [http://www2.econ.iastate.edu/tesfatsi/AMESMarketHome.htm AMES framework] example. Agent can be simple-programmed, autonomous or human-like <ref name=Chen>
Chen,S.-H.,Varieties of agentsinagent-based computational economics: A historical and an
interdisciplinary perspective. Journal of Economic Dynamics and Control(2011), doi:10.1016/j.jedc.2011.09.003, Available from: http://www.econ.iastate.edu/tesfatsi/ACEHistoricalSurvey.SHCheng2011.pdf
</ref>
In order for agents to operate in computational worlds, methods and protocols are required. These methods and protocols enable interactions between agents themselves, between agents and the world or artificial institutions e.g. market. These protocol consits of rules for mediation between agents and serve as description of interaction between agents e.g. between market and agent.
<ref name=Tesfatsion></ref>
For example in [http://en.wikipedia.org/wiki/Double_auction double auction] model, agents may have following methods:
<pre>
getWorldEventSchedule(clock time);
getWorldProtocols (collusion, insolvency);
getMarketProtocols (posting, matching, trade, settlement);
</pre>

===Equilibriums and attractors===
Model behavior can result to various types of equilibrium and attractors. '' "System is in equilibrium if all influences acting on the system offset each other
so that the system is in an unchanging condition" ''
<ref>http://dl.acm.org/citation.cfm?id=1531270</ref>. Agent-based models can help to determine which parameters influence stability or effectiveness of the market while visualization capabilities can help to identify possible [http://www.scholarpedia.org/article/Basin_of_Attraction basins of attraction]. Parameters can be changed on different levels e.g. agent level, market level or world level. Agent may have parameters like risk aversion, market may have parameters like non-employment payment percentage etc.<ref name=Tesfatsion></ref>

[[File:derivate-follower-basin-of-attraction.png]]

==Agent types and characteristics==

Simple programmed agents are represented by simple algorithm, be it short lenght of a code or simplicity of a pseudo-random number generator agent uses <ref name=Chen />. However even simple agents can exhibit form of swarm intelligence simillar to the emergent behavior of a group ants or termites. Groups of simple agents are than capable to solve complex tasks. Even without learning capability, the agents can optimize or generate orderly movement patterns. [http://en.wikipedia.org/wiki/Stigmergy Stigmergy] can be one way to achieve this. Agents can be differentiated by position in the [http://economistsview.typepad.com/economistsview/2009/02/cognitive-hierarchy-theory.html cognitive hierarchy], where more complex agents are able to think more steps ahead than simple agents. Smarter agents can also emulate behavior of simple agents if favourable but it's not possible vica versa. Non-agent economic models often introduce simplifying assumptions e.g. that all agents are rational and homogenous (CM Macal1,2* and MJ North1,2). Humans interacting in various systems or institutuins are heterogenous and it's desirable to emulate this feature to produce more realistic behavior<ref name=Chen />.

===Learning===
In order to capture dynamic nature of real markets agents should be able to learn which means change their behavior according to the situations they encounter. (zdroj) Agents in ACE can use various types of learning algorithms. Selection of an algorithm can fundamentally influence the results of the simulation<ref name=Tesfatsion />. [[Roth-Elev]] [http://en.wikipedia.org/wiki/Reinforcement_learning reinforcement learning] algorithm is one of the possible choices:
# Initialize action propensities to an initial propensity value.
# Generate choice probabilities for all actions using current propensities.
# Choose an action according to the current choice probability distribution.
# Update propensities for all actions using the reward (profits) for the last chosen action.
# Repeat from step 2.

===Possible learning types===
There are various types of learning algorithms <ref name=TesfatsionLearning />). Here is brief summary by Leigh Tesfatsion:
# Reactive Reinforcement Learning (RL)
## Example 1: Deterministic reactive RL (e.g. Derivative-Follower)
## Example 2: Stochastic reactive RL (e.g. Roth-Erev algorithms)
# Belief-Based Learning
## Example 1: Fictitious play
## Example 2: Hybrid forms (e.g. [http://www.hss.caltech.edu/~camerer/jeth2927.pdf Camerer/Ho EWA algorithm] )
# Anticipatory Learning ([http://en.wikipedia.org/wiki/Q-learning Q-Learning])
## Evolutionary Learning ([http://en.wikipedia.org/wiki/Genetic_algorithm Genetic Algorithms] - GAs)
# Connectionist Learning ([http://en.wikipedia.org/wiki/Artificial_neural_network Artificial Neural Nets] - ANNs)

In reinforcement learning algorithms, if an action ''A'' in state ''S'' produces favourable outcomes (desired reward), tendency to choose the action A should be increased. Likewise if action ''A'' produces unfavourable results tendency to choose it should be decreased. In reactive RL agent contemplates, what action should be taken based on past events. Reactive RL can be deterministic or stochastic. In first case agent is increasing or decreasing scalar decision ''D'', he moves in the same direction until the reward level starts falling. Example of second case (Roth-Erev) is in [[#Learning|Learning]]. Belief-based learning uses reflection on past choices to determine whether different action could have led to more desirable outcome. These opportunity cost assessments are then used to choose better action now. In this type of learning, agent takes into consideration presence of other agents also making their decisions. To achieve this, agent uses probability distribution function to select best response on estimated actions of other agents<ref name=TesfatsionLearning>Leigh Tesfatsion, [http://www.econ.iastate.edu/tesfatsi/LearnAlgorithms.LT.pdf Learning Algorithms: Illustrative Examples]</ref>. Example of this can be [http://en.wikipedia.org/wiki/Matching_pennies matching pennies] game:

{|class="wikitable"
|+ align="bottom" |''Matching pennies game outcome matrix''
|rowspan="2" colspan="2";| ||
!colspan="2"|Player 2
|-
|
!Heads
!Tails
|-
!rowspan=2|Player 1
! Heads || +1, −1 || −1, +1
|-
! Tails || −1, +1 || +1, −1
|}

If agent uses anticipatory learning (or [http://en.wikipedia.org/wiki/Temporal-difference_learning temporal-difference learning]), he's trying to predict what might happen in the future, if he takes some action ''A''. Relationship between value functions is therefore recursive. For each possible state it yields the optimum total reward that can be attained by the agent over current and future times. This method requires computation of transtition, return and value functions to compute optimal policy function. These functions are dependent on time and current state. [http://en.wikipedia.org/wiki/Q-learning Q-Learning] enables to compute optimal policy function without knowing these functions. Instead it iteratively acquires the Q-values, that are afterwards stored in observation history. This history is than used to estimate Q-values for next possible action choices<ref name=TesfatsionLearning />.

'''+ doplnit Genetic a neuron sítě?'''

==Software and programming==
For elaborate overview see [http://en.wikipedia.org/wiki/Comparison_of_agent-based_modeling_software Comparison of agent-based modeling software].

{| class="wikitable"
|+ Software suitable for ACE programming or visualization
!Software name || Platform and performance || Scope || Visualization capabilities || Licence
|-
| AnyLogic || Java platform || Agent-based general purpose; distributed simulations || 2D/3D models || proprietary
|-
| NetLogo || Based on Java, suitable for smaller models || simulations of natural and social phenomena, suitable for novice modelers || 2D/3D models || freeware
|}

==Other computing methods==

Linear Equations and Iterative Methods (Currently empty)
Optimization
Nonlinear Equations
Approximation
Numerical Integration and Differentiation
Monte Carlo and Simulation Methods (Currently empty)
Quasi-Monte Carlo Methods (Currently empty)
Finite Difference Methods (Currently empty)
Projection Methods for Functional Equations (Currently empty)
Numerical Dynamic Programming (Currently empty)
Regular Perturbations of Simple Systems (Currently empty)
Regular Perturbations in Multidimensional Systems (Currently empty)
Advanced Asymptotic Methods (Currently empty)
Solution Methods for Perfect Foresight Models (Currently empty)
Solving Rational Expectations Models

==References==
<references/>

* http://bucky.stanford.edu/numericalmethods/PUBCODE/DEFAULT.HTM
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/ACETutorial.pdf
* http://www2.econ.iastate.edu/classes/econ308/tesfatsion/
* http://www2.econ.iastate.edu/tesfatsi/aemind.htm
* http://www2.econ.iastate.edu/tesfatsi/aintro.htm