Text-attributed graphs (TAGs) have recently garnered significant attention due to their rich text information for each node. Graph Neural Networks (GNNs) have been widely used to leverage the text attributes and graph topology. However, traditional GNNs use shallow text embeddings and struggle with sparse and noisy structure. The advent of Large Language Models (LLMs) has introduced powerful capabilities in reasoning and text generation, promoting recent works to integrate LLMs with GNNs. However, most existing studies focus on a specific aspect of graph data, such as node features or graph structure, without fully exploring the entire graph. In this paper, we propose EFS (Enhancing Features and Structures), a novel holistic enhancement method that enhances both node features and graph structure with LLMs, which thoroughly explores the text attributes and efficiently addresses both missing and noisy edges compared to existing methods. We first design elaborate prompt templates and adopt LLMs to generate summarized keywords, ranked predictions and corresponding explanations for enriching the node features. Then we construct a lightweight edge refiner to improve the graph structure, which utilizes the previously generated ranked predictions without additional LLM API calls. Extensive experiments on various benchmark datasets show the superior performance and efficiency of EFS compared with the state-of-the-art baselines.