Earticle

Quality of Service (QoS) support in Mobile Ad hoc NETworks (MANETs) is a challenging task due to bandwidth and delay constraints, varying channel conditions, power limitations, node mobility and dynamic topology. This paper proposes an intelligent agent based on-demand (source initiated) delay aware QoS routing scheme in MANETs by using software agents that employ neuro-fuzzy logic supported by Q-learning. The proposed scheme operates in following steps. (1) Source node uses an agent that uses Dynamic Source Routing (DSR) to find various paths, their bandwidth and packet loss rate to reach a destination. (2) Parallely, a static neuro-fuzzy agent at the source node is used to optimize membership functions of fuzzy parameters according to user delay requirement of the fuzzy inference system (FIS); also, a fuzzy Q-learning static agent at the source node is employed to optimize the consequent part of if-then rules of FIS. (3) Fuzzy static agent at the source node decides whether node on a path satisfies delay requirement for an application according to the user by considering fuzzy parameters, bandwidth, packet loss rate and delay. (4) A path with QoS satisfied nodes will be selected by fuzzy QoS static agent and (5) mobile agents are used to maintain and repair the path. The scheme has been simulated in various network scenarios to test operation effectiveness and observed that proposed scheme performs better than the existing fuzzy based DSR routing methods.

Q-learning is a kind of reinforcement learning where the agent solves the given task based on rewards received from the environment. Most research done in the field of Q-learning has focused on discrete domains, although the environment with which the agent must interact is generally continuous. Thus we need to devise some methods that enable Q-learning to be applicable to the continuous problem domain. In this paper, an extended fuzzy rule is proposed so that it can incorporate Q-learning. The interpolation technique, which is widely used in memory-based learning, is adopted to represent the appropriate Q value for current state and action pair in each extended fuzzy rule. The resulting structure based on the fuzzy inference system has the capability of solving the continuous state about the environment. The effectiveness of the proposed structure is shown through simulation on the cart-pole system.

The eligibility is used to solve the credit-assignment problem which is one of important problems in reinforcement learning. Conventional eligibilities which are accumulating eligibility and replacing eligibility make ineffective use of rewards acquired in learning process. Because only an executed action in a visited state is learned by these eligibilities. Thus, we propose a new eligibility, called the weighted eligibility with which not only an executed action but also neighboring actions in a visited state are to be learned. The fuzzy Q-learning algorithm using proposed eligibility is applied to a cart-pole balancing problem, which shows improvement of learning speed.