In the present era of big data, the challenges of data management have grown significantly. One crucial aspect is the management of data storage. As data volumes continue to expand, effective storage management becomes increasingly essential. Meanwhile, evolving hardware technologies offer various storage options, ranging from HDDs to SSDs and NVRAMs. To this end, hierarchical (multi-tier) storage systems (HSS) have emerged as a solution, organizing different storage devices hierarchically to provide various storage options. However, managing multiple storage tiers and their data, while optimizing performance and cost-efficiency, is extremely complex. In this paper, we discuss the challenges in the management of hierarchical storage system. We summarise our previous contributions on tackling these challenges, including the proposal of a reinforcement learning (RL) based data migration policy and the design of an autonomous hierarchical storage management framework HSM-RL. We also present the applications of HSM-RL in scientific data management to demonstrate its adaptability and scalability. Finally, we conclude our work to date and outline the future research plans.