Color artifacts are often found at edges due to interpolation across them. To tackle this problem, four high-order interpolants for four different directions are determined, and a weight based on the corresponding absolute directional color differences is assigned to each of them. In a smooth area, the interpolants in the four directions will have equal normalized weights of 1/4. In a nonsmooth area with a sharp edge, the normalized weights for the interpolants on the opposite side of an edge will be small and less than 1/4, but this inclusion will still manifest itself as color artifacts in the output image. In our proposed method, the frequency distribution of the weights for the entire image is first determined. To reduce color artifacts, the weights are hard thresholded such that any weights below a threshold are set to zero, and that optimal threshold is determined according to the frequency distribution of the directional color differences. A detail-preserving median based multishell filter is then applied to the fully demosaicked green channel to prune outliers resulting from high-order interpolation of thin features. Based on the experimental results, it has been shown that our proposed algorithm outperforms the other algorithms with minimal color artifacts for various types of images.