Autonomous IoT systems require the development of good automation algorithms capable of handling a huge number of IoT devices such as in smart cities. Deep Reinforcement Learning (DRL) is a powerful automation technique that can be used in massive systems thanks to its ability to deal with big state spaces. Moreover, it adapts quickly to changes in the system by reinforcement learning, making the automation algorithm very flexible. However, using DRL relies generally on centralized agent architecture making it more exposed to communication failures. In this paper, we propose a distributed architecture to solve the task offloading problem in autonomous IoT systems where learning is achieved in a master agent while decision making is delegated to IoT devices. This architecture is more resilient as decisions are made locally and interactions between IoT devices and the master agent are less frequent and not blocking. We tested this architecture in the ns3-gym environment and our results show very good resilience of this architecture.