Feb 092012
 

R. Sutton at. al. explained reinforcement learning in very simple terms[1]:

“Reinforcement learning is learning what to do-how to map situations to actions-so as to maximize a numerical reward signal. The learner is not told which actions to take, as in most forms of machine learning, but instead must discover which actions yield the most reward by trying them. In the most interesting and challenging cases, actions may act not only the immediate reward but also the next situation and, through that, all subsequent rewards. These two characteristics trial-and-error search and delayed reward-are the two most important distinguishing features of reinforcement learning”.

Q-learning, a reinforcement learning technique, learns an action-value function. This function provide the  expected utility of pursuing an action in a given state and following a fixed policy afterward. In simpler terms, an agent using Q-learning learns a mapping for which action he should take when he is in one of the states of the environment. This mapping can be viewed as a table, called a Q-table, with rows as states of the agent and columns as all the actions an agent can perform in its environment. Values of each cell in a Q-table signify how favorable an action is given that an agent is in particular state. Therefore, an agent selects the best known action, depending on his current state: arg max Q(s; a).

Every action taken by an agent affects the environment, which may result in a change of the current state for the agent. Based on his action, the agent gets a reward (a real or natural number) or punishment(a negative reward). These rewards are used by the agent to learn. The goal of an agent is to maximize the total reward which he achieves, by learning the actions which are optimal for each state. Hence, the function which calculates quality of state-action combination is given by :

Q : S X A -> R

Initially, random values are set in the Q-table. Thereafter, each time an agent takes an action; a reward is given to agent, which in turn is used to update the values in Q-table. The formula for updating the Q-table is given by:

Q  Learning Fomula

The major advantages of using Q-learning are that it is simple, and it support dynamic online learning.

References

[1]  R. Sutton and A. Barto. Reinforcement Learning:An Introduction. The MIT Press Cambridge, Massachusetts London, England

 

HUMAN mind has a limited capability for processing a huge amount of detailed information in his environment; thus, to compensate, the brain groups together the information it perceives by its similarity, proximity, or functionality and assigns to each group a name or a “word” in natural language. This classification of information allows human to perform complex tasks and make intelligent decisions in an inherently vague and imprecise environment without any measurements or computation. Inspired by this human capability, Zadeh introduced the machinery of CW as a tool to formulate human reasoning with perceptions drawn from natural language and argued that the addition of CW theory to the existing tools gives rise to the theories with enhanced capabilities to deal with real-world problems and makes it possible to design systems with higher level of machine intelligence [1][2]. To do this, CW offers two principal components, (1) a language for representing the meaning of words taken from natural language, this language is called the Generalized Constraint Language (GCL), and (2) a set of deduction rules for computing and reasoning with words instead of numbers. CW is rooted in fuzzy logic; however, it offers a much more general methodology for fusion of natural language propositions and computation with fuzzy variables. CW inference rules are drawn from various fuzzy domains, such as fuzzy logic, fuzzy arithmetic, fuzzy probability, and fuzzy syllogism. This paper reports a preliminary work on the implementation of a CW inference system on top of JESS expert system shell (CWJess) . The CW reasoning is fully integrated with JESS facts and inference engine and allows knowledge to be specified in terms of GCL assertions.

Read more

 

The aim of intelligent techniques, termed game AI, used in computer video games is to provide an interesting and challenging game play to a game player. Being highly sophisticated, these games present game developers with similar kind of requirements and challenges as faced by academic AI community. The game companies claim to use sophisticated game AI to model artificial characters such as computer game bots, intelligent realistic AI agents. However, these bots work via simple routines pre-programmed to suit the game map, game rules, game type, and other parameters unique to each game. Mostly, illusive intelligent behaviors are programmed using simple conditional statements and are hard-coded in the bots’ logic. Moreover, a game programmer has to spend considerable time configuring crisp inputs for these conditional statements. Therefore, we realize a need for machine learning techniques to dynamically improve bots’ behavior and save precious computer programmers’ man-hours. We selected Qlearning, a reinforcement learning technique, to evolve dynamic intelligent bots, as it is a simple, efficient, and online learning algorithm. Machine learning techniques such as reinforcement learning are known to be intractable if they use a detailed model of the world, and also require tuning of various parameters to give satisfactory performance. Therefore, this paper examine Qlearning for evolving a few basic behaviors viz. learning to fight, and planting the bomb for computer game bots. Furthermore, we experimented on how bots would use knowledge learned from abstract models to evolve its behavior in more detailed model of the world.

Read more

 

In modern computer games, ‘bots’ – Intelligent realistic agents play a prominent role in success of a game in market. Typically, bots are modeled using finitestate machine and then programmed via simple conditional statements which are hard-coded in bots logic. Since these bots have become quite predictable to an experienced games player, she might lose her interest in game. We present a model of bots using BDI agents, which will show more human-like behavior, more believable and will provide more realistic feel to the game. These bots will use the inputs from actual game players to specify her Beliefs, Desires, and Intentions while game playing.

Read more.

 

Although simple, URL rewrite in Joomla can be pain in neck. In order for Joomla URL rewrite to work apache should be configure to allow URL rewrite in .htaccess . If you are not sure, contact your web server administrator. You can even test yourself, see http://www.addedbytes.com/for-beginners/url-rewriting-for-beginners/ .

If you are managing the server then manually enable it. If you are not sure then follow the instruction on following link see: http://www.jarrodoberto.com/articles/2011/11/enabling-mod-rewrite-on-ubuntu

Now go to Joomla installation folder and rename htaccess.txt to .htaccess

mv htaccess.txt .htaccess

Open .htaccess and remove the comment following line (find the following and remove # in front of it):

RewriteBase  /

If your Joomla is not installed in your root folder then:

RewriteBase/<Joomla Directory>

Now go to global configuration in the admin panel and enable:

Search Engine Friendly URLs
Use URL rewriting

If necessary restart the apache server. Try navigating your website it should not be showing index.php.

 

1. Introduction

Since the evident of Fuzzy logic, it has been applied to several applications such as feature detection, control theory, classification, and clustering. Fuzzy logic provides approximation methods for dealing for dealing with information that is approximate. Such information does not have well-defined boundaries. Humans can effectively deal with the information with imprecision and vague boundaries.

Traditionally, such approximate information is dealt with precision, which can be computationally expensive. Fuzzy logic provides mathematical tools for dealing with imprecise information in the manner similar to humans.

While defining AI for computer games it could be better if the linguistic terms are define in a fashion similar to humans. Expert knowledge can be easily transformed in to a machine understandable language [1]. For example, consider a speed of a vehicle, it can be slow, medium, or fast. Suppose if the range or universe of discourse of the vehicle is between 0 to 120 mph, then sets for the speed of vehicle can be define using sets such as:

Slow = {0, 1, 2, …, 40}
Medium = {41, 42, …, 80}
Fast = {81, 82, …, 120}

One way of graphically displaying these crisp sets is by considering membership of an element as 1 if an element belongs to a set and 0 otherwise. Figure 1 shows the representation of these crisp sets.

Crisp Set

Figure 1. Example of classical Set

It is obvious that speed of 90 or above can be considered as fast, and speed lower than 30 is slow. According to sets in figure 1, if we consider the borders someone driving at speed of 80 will be considered as driving with medium speed, and 81 will be considered driving fast. Moreover, different experts might now even agree with this numbers. It will be even difficulty to define rules with the hard boundaries. Fuzzy sets allow different elements with varying degree of membership, and fuzzy rules can be defined using these sets for decision making. Fuzzy system allows the translation of vague terms such as distance, and speed into in to system inputs and outputs [2].
In rest of the paper, we present an overview of fuzzy system demonstrating its application to game AI.

2. Fuzzy Sets

In classical set theory, an object is an element of a set or it is not. Fuzzy sets allow every object to be an element of a set, but with varying degree of membership. Consider a fuzzy set Speed in figure 2, speed of 81 can belong to both the sets Medium and Fast, but it belongs to the set Medium with membership 0.3 and Fast with membership 0.7. Speed of 81 is considered more to be fast then it can be considered medium. While speed of 5 will have definite membership value 1 in set Slow, and membership value 0 in Medium and Fast, meaning speed of 5 is definitely slow, and cannot be considered either Medium or Fast.

Fuzzy Set

Figure 2. Example of Fuzzy Set

Let U be a classical universal set, called universe of discourse. A fuzzy set A is described by a membership function μA(x) mapping every element of U to the unit interval [0 1].

A = {(x, μA(x)) | x ∈ U, μA(x) ∈ [0, 1]}

Core of the fuzzy sets are the membership function. Membership functions can be defined in multiple manners. Various types of membership functions are triangular, trapezoid, and s-curve.

Again consider sets representing seed of a vehicle. Figure 2 graphically represents fuzzy sets Slow, Medium, and Fast represented by left shoulder, trapezoid, and right shoulder membership functions respectively.

3. Fuzzy Linguistic Variable

A fuzzy linguistic variable is a collection of fuzzy set which represents a similar concept. For example, fuzzy sets Slow, Medium, and Fast represents linguistic variable Speed.

 Speed = {Slow, Medium, Fast}

This is more intuitive representation for a human. The information, such as speed of vehicle is slow, is efficiently encoded in digital form. Consider a fuzzy linguistic variable Distance with universe range from 0 to 500 which signifies distance from a target. Distance encapsulates fuzzy sets Far, Medium, Close. Figure 4 shows the membership function of fuzzy sets in the linguistic variable Distance.

Fuzzy Set Distance

Figure 4. Fuzzy Linguistic Variable Distance

It would be easy for a knowledge engineer to specify distance with fuzzy sets, such as Distance is Close, compared to specifying distance in units, such as Distance is 10 KM. It would be even easier to specify rules using such data structures.

4. Fuzzy Rules

Fuzzy rules are set of if-then rules with antecedents and consequents as fuzzy sets in fuzzy linguistic variable.
IF antecedent THEN consequent

Consider a scene where we need to control the speed of a boat trying to reach the target location. Above mentioned linguistic variables, Speed and Distance, can be used to define rules which will control the speed of a boat. Such rules can be defined as following:

Rule 1: IF Distance is Near THEN Speed Slow
Rule 2: IF Distance is Medium THEN Speed Medium
Rule 3: IF Distance is Far THEN Speed Fast

In the same manner rules with multiple antecedents and multiple consequent can be defined. We can determine the speed that needs to be set given the distance of the boat from the target.

5. Fuzzy Rule Inference

Fuzzy rule inference is a three step process:

  1. Fuzzification: determining the membership value of the crisp input in the different fuzzy sets of linguistic variable
  2. Fuzzy rules: apply fuzzy rules to the fuzzified input in order to determine the fuzzy output.
  3. Defuzzification: usually fuzzy output is converted back to a crisp value, especially for game, to be used in the application

Consider the example of controlling the speed of a boat. Suppose that current distance of boat is about 25 miles from target. Figure 4 shows the fuzzification of the input (25). Values 25 belong to Near and Medium fuzzy set with membership 0.8 and 0.2 respectively. Although not shown in the figure 5, values 25 also belong to fuzzy set Far, but with membership value 0.

Next step is to apply all the fuzzy rules. For each rule determine the confident in the consequent of the rule based on antecedent of the rules. In our example there is only one antecedent, therefore the confidence if the membership value of the input. For rule 1 with antecedent Near confidence for the input will 0.8 for Speed Slow. Similarly, for rule 2 confidences in Medium is 0.2, and 0.0 for Far in rule 3.

Fuzzy Rules Inference Example

Figure 4. Fuzzy rule inference

Defuzzification

Figure 5. Defuzzification

Based on the level of confidence each consequent is clipped and then combined in to a single fuzzy set as shown in figure 5. Finally, the output fuzzy set is defuzzified for getting the crisp value. A standard method for defuzzification is to calculate the center of gravity of the output fuzzy set. As shown in figure 5, this would yield the speed of a boat which is approximately 20 mph.

Even if there is more than one antecedent we still follow the same process, except after fuzzification in order to determine the level of confidence we take into consideration the operator between the antecedents. If there is an AND operator between the antecedents then minimum of the membership value is taken as level of confidence, or if there it is an OR operator then maximum of all the membership value if taken as level of confidence.

Other examples of the fuzzy logic usage in games can be found in [1] and [3].

7. Conclusion

Fuzzy Logic provides a flexible and powerful tool for handling and processing approximate information. Fuzzy system model reduces the time and effort of game programmers require for configuring crisp values. Also, the number of rules needed in a game for decision making can be significantly reduced using a fuzzy system.

Future work should include incorporating fuzzy rules with an expert system. Expert systems are widely used in industries for simplifying complex decision making processes. A fuzzy expert system can give more power of expression to experts and can handle different form of inputs from different experts.

8. References

[1] M. Buckland. Programming game AI by example, Wordware Publishing, Inc., 2005, pp. 415-456
[2] Adnan Shaout , Brady King , Luke Reisner . Real-Time Game Design of Pac-Man Using Fuzzy Logic. The International Arab Journal of Information Technology (IAJIT), 2005.
[3] Ian Millington, and John Funge. Artificial Intelligence for Games, Second Edition. Elsevier Inc . pp. 371-395 2009.

Note: Extracted from CS 487 Fall 2011 class presentation report. 

 

Create a directory for eclipse.

~ $ mkdir  ~/opt

Download eclipse from http://www.eclipse.org/downloads/.

Open terminal and cd to the download directory.  Extract files, and move them to ~/opt directory.

~ $ tar -zxvf eclipse-java-indigo-SR1-linux-gtk-x86_64.tar.gz && mv eclipse ~/opt

Create an entry in /usr/bin to launch eclipse.

~ $ sudo nano /usr/bin/eclipse 

Copy following lines, save and close the file.

export MOZILLA_FIVE_HOME="/usr/lib/mozilla/"
export ECLIPSE_HOME="$HOME/opt/eclipse"

$ECLIPSE_HOME/eclipse $*

Make it executable

sudo chmod +x /usr/bin/eclipse

Type eclipse in terminal to run it.

~ $ eclipse

Create an launcher on desktop:

~ $ nano ~/Desktop/eclipse.desktop

Copy following lines.

#!/usr/bin/env xdg-open

[Desktop Entry]
 Version=1.0
 Type=Application
 Terminal=false
 Icon[en_US]=eclipse
 Name[en_US]=eclipse
 Exec=eclipse
 Name=eclipse
 Icon=eclipse

Save and close.

 

First make sure you have Java Installed on your system.

sudo apt-get install  openjdk-6-jdk

Download Netbeans from http://netbeans.org/index.html .Open terminal and cd to the download location

cd ~/Downloads
Start the installer and follow the on screen instruction. On terminal type:
sh netbeans-7.0.1-ml-linux.sh
 

If you get following error in eclipse:

Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /usr/lib/jni/

Close eclipse and run following command on terminal.

sudo apt-get install libsvn-java

Edit eclipse.ini file.

sudo nano /usr/lib/eclipse/eclipse.ini

Add following two lines after -vmargs

-Djava.library.path=/usr/share/java/
-Djava.library.path=/usr/lib/jni/

save and close

If you are still getting the same error, this is because Ubuntu 11.10 in lagging in JavaHL version. If you install from Ubuntu 11.10 repository , it installs 1.6.x, although the newer version is 1.7.x . Simple solution would be to uninstall subversion 1.8.x from eclipse and install version 1.6.x.

See:

http://subclipse.tigris.org/servlets/ProjectProcess?pageID=p4wYuA

http://subclipse.tigris.org/wiki/JavaHL

 

Nov 162011
 

 Application

Package Names

 SSH server openssh-server
 SSH client openssh-client
 Open JDK 6  openjdk-6-jdk